Systems and methods for deep learning-based subject persistence

ABSTRACT

Systems and techniques are provided for tracking subjects in an area of real space. The system receives a plurality of sequences of images of corresponding fields of view in the real space. The system uses the sequences of images to locate subjects in the area of real space during identification intervals. The system compares a current set of located subjects from a current identification interval with previous sets of located subjects from a plurality of previous identification intervals to track locations of tracked subjects in the area.

BACKGROUND Field

The present invention relates to systems that track subjects in an areaof real space.

Description of Related Art

A difficult problem in image processing arises when images of subjectsfrom cameras are used to identify and track subjects in an area of realspace such as a shopping store. The system needs to keep track ofsubjects in the area of real space for the duration of subject'spresence. The subjects can leave the area of real space withoutcommunicating with the system. The system may also lose track of asubject due to other reasons. For example, when a subject is temporarilyobscured, or moves to a location in the area of real space that is notin the field of view of cameras or sensors used to track the subjects ordue to errors in the system.

As new subjects are located in the area of real space, the system needsto determine whether a new subject is a tracked subject who went missingfor a period of time or a new subject who has recently entered the areaof real space.

It is desirable to provide a system that can more effectively andautomatically track subjects in the area of real space.

SUMMARY

A system, and method for operating a system, are provided for trackingsubjects, in an area of real space. The system receives a plurality ofsequences of images of corresponding fields of view in the real space.The system comprises a processing system which include logic that usesthe sequences of images to locate subjects in the area of real spaceduring identification intervals. The system includes logic that comparesa current set of located subjects from a current identification intervalwith previous sets of located subjects from a plurality of previousidentification intervals to track locations of tracked subjects in thearea.

In one embodiment the processing system has access to a database storinga set of identified subjects (e.g., checked in, or otherwise linked toan account). The processing system includes logic to link trackedsubjects with identified subjects. If a currently located subject cannotbe linked with a tracked subject from previous identification intervals,then a procedure to link currently located subject to an account, suchas a check-in routine, must be executed. Utilizing persistence logic asdescribed herein, with more robust tracking persistence, the needexecute repeated operations to link located subjects to accounts can bereduced.

In one embodiment, the logic that compares a current set of locatedsubjects from a current identification interval with previous sets oflocated subjects from a plurality of previous identification intervalsincludes logic that detects a condition in which a number of locatedsubjects in the current set does not match the number of locatedsubjects from a first preceding identification interval in the pluralityof previous intervals. Upon detection of the condition, the systemcompares at least one of the located subjects in the current set withthe set of located subjects from a second preceding identificationinterval in the plurality of previous identification intervals, thatprecedes the first preceding identification interval.

In one embodiment, upon detection of the condition, if the set oflocated subjects from the first preceding identification intervalincludes N members, and the set of located subjects from the secondpreceding identification interval includes N members plus a missingtracked subject, and the current set of located subjects includes Nmembers plus a candidate located subject. Then the system links thecandidate located subject located from the current identificationinterval with the missing tracked subject located from the secondpreceding identification interval.

In such an embodiment, the linking the candidate located subject locatedfrom the current identification interval with the missing trackedsubject located from the second preceding identification intervalincludes calculating difference between their respective locationtimestamps. The candidate located subject and the missing trackedsubject are linked if a difference in a timestamp of time of location ofthe candidate located subject and the missing tracked subject is lessthan a first threshold.

In another embodiment, the linking the candidate located subject locatedfrom the current identification interval with the missing trackedsubject located from the second preceding identification intervalincludes calculating distance between their respective locations in thearea of real space. The candidate located subject and the missingtracked subject are linked if a distance between locations of thecandidate located subject and the missing tracked subject is less than asecond threshold.

In another embodiment, the linking the candidate located subject locatedfrom the current identification interval with the missing trackedsubject located from the second preceding identification intervalincludes calculating their respective distances to a location in thearea of real space also referred to as a sink (or the boundary of anexit or entrance region). The candidate located subject and the missingtracked subject are linked if a distance between their respectivelocations and the location of the area of real space is less than athird threshold.

In one embodiment, the system locates more than one candidate locatedsubjects in a current identification interval (also referred to as thirdidentification interval). In such an embodiment, the system includes thefollowing logic to match located subjects in the current identificationinterval with tracked subjects in the preceding identification intervaland update the tracked subjects. If the set of located subjects from thefirst preceding identification interval includes N members that aretracked, and the set of located subjects from the current identificationinterval includes N members plus more than one candidate locatedsubjects. Then the system includes logic to simplify linking N membersof the set of located subjects from the current identification intervalto N members of the set of tracked subjects from the first precedingidentification interval. The system prioritizes members of the set oflocated subjects from the current identification interval to identify aset of N located subjects to link to the set N tracked subjects from thefirst preceding identification interval.

In such an embodiment, the prioritizing the members of the set oflocated subjects from the current identification interval to identifythe set of N located subjects includes logic that calculates distancebetween pairs of N members plus more than one candidate located subjectslocated from the current identification interval. The system thenidentifies the set of N located subjects by comparing the calculateddistance with a second threshold.

In such an embodiment, the system includes logic that links a member ofthe set of N tracked subjects from the first preceding identificationinterval to a member of the set of N located subjects in the currentidentification interval. The system calculates distances betweenlocations of the member of the set of N tracked subjects from the firstpreceding identification interval and the member of the set of N locatedsubjects in the current identification interval. The system compares thecalculated distances with a second threshold.

The system includes logic that links a member of the set of N trackedsubjects from the first preceding identification interval to a member ofthe set of N located subjects in the current identification interval.The system calculates distances between locations of the member of theset of N tracked subjects from the first preceding identificationinterval and the member of the set of N located subjects in the currentidentification interval to a sink i.e., boundary of an exit or entranceregion. The system compares the calculated distances with a thirdthreshold.

The system links the member of the set of N tracked subjects from thefirst preceding identification interval to the member of the set of Nlocated subjects in the current identification interval by determiningthat no other member of the set of N tracked subjects from the firstpreceding identification interval matches the member of the set of Nlocated subjects in the current identification interval.

The system includes logic that calculates distance between location ofthe missing tracked subject and a designated unmonitored location in thearea of real space and logic that calculates distance between locationof the candidate located subject and the designated unmonitored locationin the area of real space. The system compares the calculated distanceswith a third threshold. The system matches the candidate located subjectfrom the current identification interval with the missing trackedsubject by determining that no other subject in the set of trackedsubjects in the second preceding identification interval and the set oflocated subjects in the current identification interval is closer to thedesignated unmonitored location than the third threshold.

In one embodiment, the plurality of sequences of images of correspondingfields of view in the area of real space are produced by a plurality ofsensors in the area of real space producing synchronized sequences ofimages of respective fields of view. In one embodiment, the sensorscomprise cameras.

In one embodiment, one image in each of the plurality of sequences iscaptured in each image capture cycle, and identification intervalscorrespond with the image capture cycles. In another embodiment, theidentification interval comprises more than one image capture cycles.

In one embodiment, the logic that uses the sequences of images to locatesubjects in the area of real space during identification intervalsincludes locates a subject as a constellation of joints in the area ofreal space, a joint in the constellation of joints including a locationrepresented by positions in three dimensions of the area of real spaceand a timestamp. In one embodiment, the timestamp corresponds with thetime at which the image is captured in the image capture cycle.

Also described herein is a system a processing system receiving aplurality of sequences of images of corresponding fields of view in thereal space, including logic that uses the sequences of images to locatesubjects in the area of real space during identification intervals;logic that matches located subjects from a plurality of identificationintervals to identify tracked subjects; logic that links trackedsubjects to user accounts, to maintain a record of identified subjectsin the area of real space; and logic that compares located subjects in acurrent identification interval with identified subjects in a pluralityof preceding identification intervals to update locations of theidentified subjects, including when there is a mismatch in numbers oflocated subjects in the current identification interval with numbers ofidentified subjects in a first preceding identification interval, thencomparing at least one of the located subjects in the currentidentification interval with identified subjects in at least one moreother preceding identification interval.

In embodiments describe herein, the logic to link tracked subjects withuser accounts operates without personal biometric information aboutusers.

Also, in embodiments described herein, the logic to link trackedsubjects with user accounts includes a procedure to identify locationsof mobile devices executing client applications in the area of realspace, and matching locations of the mobile devices with locations ofthe tracked or located subjects.

Methods and computer program products which can be executed by computersystems are also described herein.

Functions described herein, including but not limited to identifying andtracking a candidate located subject at a particular identificationinterval in an area of real space and to matching the candidate locatedsubject to the missing tracked subject located at differentidentification intervals and data structure configured for use to matchthe candidate located subject to the missing tracked subject presentcomplex problems of computer engineering, relating for example to thetype of image data to be processed, what processing of the image data toperform, and how to determine actions from the image data with highreliability.

Other aspects and advantages of the present invention can be seen onreview of the drawings, the detailed description and the claims, whichfollow.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an architectural level schematic of a system in whicha subject persistence processing engine tracks subjects in an area ofreal space.

FIG. 2A is a side view of an aisle in a shopping store illustrating asubject, inventory display structures and a camera arrangement in ashopping store.

FIG. 2B is a perspective view, illustrating a subject taking an itemfrom a shelf in the inventory display structure in the area of realspace.

FIG. 3A shows an example data structure for storing joints informationof subjects.

FIG. 3B is an example data structure for storing a subject including theinformation of associated joints.

FIG. 4A shows tracked subjects in an area of real space in a secondpreceding identification interval.

FIG. 4B shows tracked subjects in an area of real space in a firstpreceding identification interval in which one tracked subject locatedin the second preceding identification interval is missing.

FIG. 4C shows subjects located in an area of real space in a currentidentification interval in which a candidate subject is located.

FIG. 5 is an example flow chart for matching a candidate located subjectto a missing tracked subject.

FIG. 6A shows tracked subjects in the area of real space located in afirst preceding identification interval.

FIG. 6B shows subjects located in the area of real space in a currentidentification interval with more than one located subjects not matchedwith tracked subjects located in a first preceding identificationinterval.

FIG. 7 is an example flow chart illustrating process steps for matchingsubjects located in the current identification interval to trackedsubjects in the first preceding identification interval when more thanone located subjects in the current identification interval are notmatched with any tracked subject in the first preceding identificationinterval.

FIG. 8A shows an area of real space with a designated unmonitoredlocation and a tracked subject located in a second precedingidentification interval, positioned close to the designated unmonitoredlocation.

FIG. 8B shows the area of real space with tracked subjects located in afirst preceding identification interval and the tracked subject of FIG.8A positioned close to the designated unmonitored location missing inthe first preceding identification interval.

FIG. 8C shows subjects located in the current identification interval inthe area of real space including a candidate located subject positionedclose to the designated unmonitored location.

FIG. 9 is an example flow chart with process steps to match thecandidate located subject close to the designated unmonitored locationto a missing tracked subject.

FIG. 10 is a camera and computer hardware arrangement configured forhosting the subject persistence processing engine of FIG. 1.

FIG. 11 is a side view of an aisle in a shopping store illustrating asubject with a mobile computing device and a camera arrangement.

FIG. 12 is a top view of the aisle of FIG. 11 in a shopping storeillustrating the subject with the mobile computing device and the cameraarrangement.

FIG. 13 is a flowchart showing process steps for identifying a subjectby matching the tracked subject to a user account using a semaphoreimage displayed on a mobile computing device.

FIG. 14 is a flowchart showing process steps for identifying a subjectby matching a tracked subject to a user account using service locationof a mobile computing device.

FIG. 15 is a flowchart showing process steps for identifying a subjectby matching a tracked subject to a user account using velocity ofsubjects and a mobile computing device.

FIG. 16A is a flowchart showing a first part of process steps formatching a tracked subject to a user account using a network ensemble.

FIG. 16B is a flowchart showing a second part of process steps formatching a tracked subject to a user account using a network ensemble.

FIG. 16C is a flowchart showing a third part of process steps formatching a tracked subject to a user account using a network ensemble.

FIG. 17 is an example architecture in which the four techniquespresented in FIGS. 13 to 16C are applied in an area of real space toreliably match a tracked subject to a user account.

DETAILED DESCRIPTION

The following description is presented to enable any person skilled inthe art to make and use the invention, and is provided in the context ofa particular application and its requirements. Various modifications tothe disclosed embodiments will be readily apparent to those skilled inthe art, and the general principles defined herein may be applied toother embodiments and applications without departing from the spirit andscope of the present invention. Thus, the present invention is notintended to be limited to the embodiments shown but is to be accordedthe widest scope consistent with the principles and features disclosedherein.

System Overview

A system and various implementations of the subject technology aredescribed with reference to FIGS. 1-17. The system and processes aredescribed with reference to FIG. 1, an architectural level schematic ofa system in accordance with an implementation. Because FIG. 1 is anarchitectural diagram, certain details are omitted to improve theclarity of the description.

The discussion of FIG. 1 is organized as follows. First, the elements ofthe system are described, followed by their interconnections. Then, theuse of the elements in the system is described in greater detail.

FIG. 1 provides a block diagram level illustration of a system 100. Thesystem 100 includes cameras 114, network nodes hosting image recognitionengines 112 a, 112 b, and 112 n, mobile computing devices 118 a, 118 b,118 m (collectively referred as mobile computing devices 120), a subjectpersistence processing engine 180 deployed in a network node 104 (ornodes) on the network, a network node 102 hosting a subject trackingengine 110, and a matching engine 170 deployed in a network node ornodes (also known as a processing platform) 103. The network nodes caninclude or have access to memory supporting subject tracking andpersistence, including in this example a maps database 140, a subjectsdatabase 150, a persistence heuristics database 160, a training database162, a user account database 164, an image database 166, and acommunication network or networks 181. The network nodes can host onlyone image recognition engine, or several image recognition engines.

The embodiment described here uses cameras in the visible range whichcan generate for example RGB color output images. In other embodiments,different kinds of sensors are used to produce sequences of images.Examples of such sensors include, ultrasound sensors, thermal sensors,Lidar, etc., which are used to produce sequences of images ofcorresponding fields of view in the real space. In one embodiment,sensors can be used in addition to the cameras 114. Multiple sensors canbe synchronized in time with each other, so that frames are captured bythe sensors at the same time, or close in time, and at the same framecapture rate. All of the embodiments described herein can includesensors other than or in addition to cameras.

As used herein, a network node is an addressable hardware device orvirtual device that is attached to a network, and is capable of sending,receiving, or forwarding information over a communications channel to orfrom other network nodes. Examples of electronic devices which can bedeployed as hardware network nodes include all varieties of computers,workstations, laptop computers, handheld computers, and smartphones.Network nodes can be implemented in a cloud-based server system. Morethan one virtual device configured as a network node can be implementedusing a single physical device.

The databases 140, 150, 160, 162, 164, and 166 are stored on one or morenon-transitory computer readable media. As used herein, no distinctionis intended between whether a database is disposed “on” or “in” acomputer readable medium. Additionally, as used herein, the term“database” does not necessarily imply any unity of structure. Forexample, two or more separate databases, when considered together, stillconstitute a “database” as that term is used herein. Thus in FIG. 1, thedatabases 140, 150, 160, 162, 164, and 166 can be considered to be asingle database.

For the sake of clarity, only three network nodes hosting imagerecognition engines are shown in the system 100. However, any number ofnetwork nodes hosting image recognition engines can be connected to thesubject tracking engine 110 through the network(s) 181. Similarly, theimage recognition engine, the subject tracking engine, the subjectpersistence processing engine, the matching engine and other processingengines described herein can execute using more than one network node ina distributed architecture.

The interconnection of the elements of system 100 will now be described.Network(s) 181 couples the network nodes 101 a, 101 b, and 101 n,respectively, hosting image recognition engines 112 a, 112 b, and 112 n,the network node 104 hosting the subject persistence processing engine180, the network node 102 hosting the subject tracking engine 110, thenetwork node 103 hosting the matching engine 170, the maps database 140,the subjects database 150, the persistence heuristics database 160, thetraining database 162, the user database 164, the image database 166 andthe mobile computing devices 120. Cameras 114 are connected to thesubject tracking engine 110 through network nodes hosting imagerecognition engines 112 a, 112 b, and 112 n. In one embodiment, thecameras 114 are installed in a shopping store such that sets of cameras114 (two or more) with overlapping fields of view are positioned tocapture images of real space in the store. In FIG. 1, two cameras arearranged over aisle 116 a, two cameras are arranged over aisle 116 b,and three cameras are arranged over aisle 116 n. Cameras are installedover open spaces, aisles, and near exits and entrances to the shoppingstore. In such an embodiment, the cameras are configured with the goalthat customers moving in the shopping store are present in the field ofview of two or more cameras at any moment in time.

Cameras 114 can be synchronized in time with each other, so that imagesare captured at the image capture cycles at the same time, or close intime, and at the same image capture rate. The cameras 114 can sendrespective continuous streams of images at a predetermined rate tonetwork nodes hosting image recognition engines 112 a-112 n. Imagescaptured in all the cameras covering an area of real space at the sametime, or close in time, are synchronized in the sense that thesynchronized images can be identified in the processing engines asrepresenting different views of subjects having fixed positions in thereal space. For example, in one embodiment, the cameras send imageframes at the rates of 30 frames per second (fps) to respective networknodes hosting image recognition engines 112 a-112 n. Each frame has atimestamp, identity of the camera (abbreviated as “camera_id”), and aframe identity (abbreviated as “frame_id”) along with the image data. Asdescribed above other embodiments of the technology disclosed can usedifferent types of sensors such as image sensors, ultrasound sensors,thermal sensors, Lidar, etc.,

Cameras are connected to respective image recognition engines. Forexample, in FIG. 1, the two cameras installed over the aisle 116 a areconnected to the network node 101 a hosting an image recognition engine112 a. Likewise, the two cameras installed over aisle 116 b areconnected to the network node 101 b hosting an image recognition engine112 b. Each image recognition engine 112 a-112 n hosted in a networknode or nodes 101 a-101 n, separately processes the image framesreceived from one camera each in the illustrated example. In anembodiment of a subject tracking system described herein, cameras areinstalled overhead, so that in combination the fields of view of thecameras encompass an area of real space in which the tracking is to beperformed, such as in a shopping store.

In one embodiment, each image recognition engine 112 a, 112 b, and 112 nis implemented as a deep learning algorithm such as a convolutionalneural network (abbreviated CNN). In such an embodiment, the CNN istrained using training database. In an embodiment described herein,image recognition of subjects in the real space is based on identifyingand grouping features of the subjects such as joints, recognizable inthe images, where the groups of joints can be attributed to anindividual subject. For this joints-based analysis, the trainingdatabase has a large collection of images for each of the differenttypes of joints for subjects. In the example embodiment of a shoppingstore, the subjects are the customers moving in the aisles between theshelves. In an example embodiment, during training of the CNN, thesystem 100 is referred to as a “training system.” After training the CNNusing the training database, the CNN is switched to production mode toprocess images of customers in the shopping store in real time.

In an example embodiment, during production, the system 100 is referredto as a runtime system (also referred to as an inference system). TheCNN in each image recognition engine produces arrays of joints datastructures for images in its respective stream of images. In anembodiment as described herein, an array of joints data structures isproduced for each processed image, so that each image recognition engine112 a-112 n produces an output stream of arrays of joints datastructures. These arrays of joints data structures from cameras havingoverlapping fields of view are further processed to form groups ofjoints, and to identify such groups of joints as subjects. The subjectscan be tracked by the system using a tracking identifier “tracking_id”during their presence in the area of real space. The tracked subjectscan be saved in the subjects database 150.

The subject tracking engine 110, hosted on the network node 102receives, in this example, continuous streams of arrays of joints datastructures for the subjects from image recognition engines 112 a-112 n.The subject tracking engine 110 processes the arrays of joints datastructures identified from the sequences of images received from thecameras at image capture cycles. It then translates the coordinates ofthe elements in the arrays of joints data structures corresponding toimages in different sequences into candidate joints having coordinatesin the real space. For each set of synchronized images, the combinationof candidate joints identified throughout the real space can beconsidered, for the purposes of analogy, to be like a galaxy ofcandidate joints. For each succeeding point in time, movement of thecandidate joints is recorded so that the galaxy changes over time. Theoutput of the subject tracking engine 110 is used to locate subjects inthe area of real space during identification intervals. One image ineach of the plurality of sequences of images, produced by the cameras,is captured in each image capture cycle.

The subject tracking engine 110 uses logic to determine groups or setsof candidate joints having coordinates in real space as subjects in thereal space. For the purposes of analogy, each set of candidate points islike a constellation of candidate joints at each point in time. In oneembodiment, these constellations of joints are generated peridentification interval as representing a located subject. Subjects arelocated during an identification interval using the constellation ofjoints. The constellations of candidate joints can move over time. Atime sequence analysis of the output of the subject tracking engine 110over a period of time, such as over multiple temporally orderedidentification intervals, identifies movements of subjects in the areaof real space. The system can store the subject data including uniqueidentifiers, joints and their locations in the real space in the subjectdatabase 150.

In an example embodiment, the logic to identify sets of candidate joints(i.e. constellations) as representing a located subject comprisesheuristic functions is based on physical relationships amongst joints ofsubjects in real space. These heuristic functions are used to locatesets of candidate joints as subjects. The sets of candidate jointscomprise individual candidate joints that have relationships accordingto the heuristic parameters with other individual candidate joints andsubsets of candidate joints in a given set that has been located, or canbe located, as an individual subject.

Located subjects in one identification interval can be matched withlocated subjects in other identification intervals based on location andtiming data. Located subjects matched this way are referred to herein astracked subjects, and their location can be tracked in the system asthey move about the area of real space across identification intervals.In the system, a list of tracked subjects from each identificationinterval over some time window can be maintained, including for exampleby assigning a unique tracking identifier to members of a list oflocated subjects for each identification interval, or otherwise. Locatedsubjects in a current identification interval are processed to determinewhether they correspond to tracked subjects from one or more previousidentification intervals. If they are matched, then the location of thetracked subject is updated to the location of the current identificationinterval. Located subjects not matched with tracked subjects fromprevious intervals are further processed to determine whether theyrepresent newly arrived subjects, or subjects that had been trackedbefore, but have been missing from an earlier identification interval.

For the purposes of tracking subjects, the subject persistenceprocessing engine 180 compares the newly located subjects in the currentidentification interval with one or more preceding identificationintervals. The system includes logic to determine if the newly locatedsubject is a missing tracked subject previously tracked in an earlieridentification interval and stored in the subjects database but who wasnot matched with a located subject in an immediately precedingidentification interval. If the newly located subject in the currentidentification interval is matched to the missing tracked subjectlocated in the earlier identification interval, the system updates themissing tracked subject in the subject database 150 using the candidatelocated subject located from the current identification interval.

In one embodiment, in which the subject is represented as aconstellation of joints as discussed above, the positions of the jointsof the missing tracked subject is updated in the database with thepositions of the corresponding joints of the candidate located subjectlocated from the current identification interval. In this embodiment,the system stores information for tracked subject in the subjectdatabase 150. This can include information such as the identificationintervals in which the tracked subject is located. Additionally, thesystem can also store, for a tracked subject, the identificationintervals in which the tracked subject is not located. In anotherembodiment, the system can store missing tracked subjects in a missingsubjects database, or tag tracked subjects as missing, along withadditional information such as the identification interval in which thetracked subject went missing and last known location of the missingtracked subject in the area of real space. In some embodiments, thesubject status as tracked and located, can be stored per identificationinterval.

The subject persistence engine 180 can process a variety of subjectpersistence scenarios. For example, a situation in which more than onecandidate located subjects are located in the current identificationinterval but not matched with tracked subjects, or a situation when alocated subject moves to a designated unmonitored location in the areaof real space but reappears after some time and is located near thedesignated unmonitored location in the current identification interval.The designated unmonitored location in the area of real space can be arestroom, for example. The technology can use persistence heuristics toperform the above analysis. In one embodiment, the subject persistenceheuristics are stored in the persistence heuristics database 160.

In the example of a shopping store the customers (also referred to assubjects above) move in the aisles and in open spaces. The customerstake items from inventory locations on shelves in inventory displaystructures. In one example of inventory display structures, shelves arearranged at different levels (or heights) from the floor and inventoryitems are stocked on the shelves. The shelves can be fixed to a wall orplaced as freestanding shelves forming aisles in the shopping store.Other examples of inventory display structures include, pegboardshelves, magazine shelves, lazy susan shelves, warehouse shelves, andrefrigerated shelving units. The inventory items can also be stocked inother types of inventory display structures such as stacking wirebaskets, dump bins, etc. The customers can also put items back on thesame shelves from where they were taken or on another shelf.

In one embodiment, the image analysis is anonymous, i.e., a uniquetracking identifier assigned to a subject created through jointsanalysis does not identify personal identification details (such asnames, email addresses, mailing addresses, credit card numbers, bankaccount numbers, driver's license number, etc.) of any specific subjectin the real space. The data stored in the subjects database 150 does notinclude any personal identification information. The operations of thesubject persistence processing engine 180 and the subject trackingengine 110 do not use any personal identification including biometricinformation associated with the subjects.

In one embodiment, the tracked subjects are identified by linking themto respective “user accounts” containing for example preferred paymentmethod provided by the subject. When linked to a user account, a trackedsubject is characterized herein as an identified subject. Track subjectsare linked with items picked up on the store, and linked with a useraccount, for example, and upon exiting the store, an invoice can begenerated and delivered to the identified subject, or a financialtransaction executed on line to charge the identified subject using thepayment method associated to their accounts. The identified subjects canbe uniquely identified, for example, by unique account identifiers orsubject identifiers, etc. In the example of a cashier-less store, as thecustomer completes shopping by taking items from the shelves, the systemprocesses payment of items bought by the customer.

The system includes a matching engine 170 (hosted on the network node103) to process signals received from mobile computing devices 120(carried by the subjects) to match the identified subjects with useraccounts. The matching can be performed by identifying locations ofmobile devices executing client applications in the area of real space(e.g., the shopping store) and matching locations of mobile devices withlocations of subjects, without use of personal identifying biometricinformation from the images.

The actual communication path to the network nodes 104 hosting thesubject persistence processing engine 180 and the network node 102hosting the subject tracking engine 110, through the network 181 can bepoint-to-point over public and/or private networks. The communicationscan occur over a variety of networks 181, e.g., private networks, VPN,MPLS circuit, or Internet, and can use appropriate applicationprogramming interfaces (APIs) and data interchange formats, e.g.,Representational State Transfer (REST), JavaScript™ Object Notation(JSON), Extensible Markup Language (XML), Simple Object Access Protocol(SOAP), Java™ Message Service (JMS), and/or Java Platform Module System.All of the communications can be encrypted. The communication isgenerally over a network such as a LAN (local area network), WAN (widearea network), telephone network (Public Switched Telephone Network(PSTN), Session Initiation Protocol (SIP), wireless network,point-to-point network, star network, token ring network, hub network,Internet, inclusive of the mobile Internet, via protocols such as EDGE,3G, 4G LTE, Wi-Fi, and WiMAX. Additionally, a variety of authorizationand authentication techniques, such as username/password, OpenAuthorization (OAuth), Kerberos, SecureID, digital certificates andmore, can be used to secure the communications.

The technology disclosed herein can be implemented in the context of anycomputer-implemented system including a database system, a multi-tenantenvironment, or a relational database implementation like an Oracle™compatible database implementation, an IBM DB2 Enterprise Server™compatible relational database implementation, a MySQL™ or PostgreSQL™compatible relational database implementation or a Microsoft SQL Server™compatible relational database implementation or a NoSQL™ non-relationaldatabase implementation such as a Vampire™ compatible non-relationaldatabase implementation, an Apache Cassandra™ compatible non-relationaldatabase implementation, a BigTable™ compatible non- relational databaseimplementation or an HBase™ or DynamoDB™ compatible non- relationaldatabase implementation. In addition, the technology disclosed can beimplemented using different programming models like MapReduce™, bulksynchronous programming, MPI primitives, etc. or different scalablebatch and stream management systems like Apache Storm™, Apache Spark™,Apache Kafka™, Apache Flink™, Truviso™, Amazon Elasticsearch Service™,Amazon Web Services™ (AWS), IBM Info-Sphere™, Borealis™, and Yahoo! S4™.

Camera Arrangement

The cameras 114 are arranged to track subjects (or entities) in a threedimensional (abbreviated as 3D) real space. In the example embodiment ofthe shopping store, the real space can include the area of the shoppingstore where items for sale are stacked in shelves. A point in the realspace can be represented by an (x, y, z) coordinate system. Each pointin the area of real space for which the system is deployed is covered bythe fields of view of two or more cameras 114.

In a shopping store, the shelves and other inventory display structurescan be arranged in a variety of manners, such as along the walls of theshopping store, or in rows forming aisles or a combination of the twoarrangements. FIG. 2A shows an arrangement of shelf unit A 202 and shelfunit B 204, forming an aisle 116 a, viewed from one end of the aisle 116a. Two cameras, camera A 206 and camera B 208 are positioned over theaisle 116 a at a predetermined distance from a roof 230 and a floor 220of the shopping store above the inventory display structures, such asshelf units A 202 and shelf unit B 204. The cameras 114 comprise camerasdisposed over and having fields of view encompassing respective parts ofthe inventory display structures and floor area in the real space. Forexample, the field of view 216 of camera A 206 and field of view 218 ofcamera B 208 overlap as shown in FIG. 2A. The locations of subjects arerepresented by their positions in three dimensions of the area of realspace. In one embodiment, the subjects are represented as constellationof joints in real space. In this embodiment, the positions of the jointsin the constellation of joint are used to determine the location of asubject in the area of real space.

In the example embodiment of the shopping store, the real space caninclude the entire floor 220 in the shopping store. Cameras 114 areplaced and oriented such that areas of the floor 220 and shelves can beseen by at least two cameras. The cameras 114 also cover floor space infront of the shelves 202 and 204. Camera angles are selected to haveboth steep perspective, straight down, and angled perspectives that givemore full body images of the customers. In one example embodiment, thecameras 114 are configured at an eight (8) foot height or higherthroughout the shopping store. In one embodiment, the area of real spaceincludes one or more designated unmonitored locations such as restrooms.

Entrances and exits for the area of real space, which act as sources andsinks of subjects in the subject tracking engine, are stored in the mapsdatabase. Also, designated unmonitored locations are not in the field ofview of cameras 114, which can represent areas in which tracked subjectsmay enter, but must return into the area being tracked after some time,such as a restroom. The locations of the designated unmonitoredlocations are stored in the maps database 140. The locations can includethe positions in the real space defining a boundary of the designatedunmonitored location and can also include location of one or moreentrances or exits to the designated unmonitored location.

Three Dimensional Scene Generation

In FIG. 2A, a subject 240 is standing by an inventory display structureshelf unit B 204, with one hand positioned close to a shelf (notvisible) in the shelf unit B 204. FIG. 2B is a perspective view of theshelf unit B 204 with four shelves, shelf 1, shelf 2, shelf 3, and shelf4 positioned at different levels from the floor. The inventory items arestocked on the shelves.

A location in the real space is represented as a (x, y, z) point of thereal space coordinate system. “x” and “y” represent positions on atwo-dimensional (2D) plane which can be the floor 220 of the shoppingstore. The value “z” is the height of the point above the 2D plane atfloor 220 in one configuration. The system combines 2D images from twoor more cameras to generate the three dimensional positions of joints inthe area of real space. This section presents a description of theprocess to generate 3D coordinates of joints. The process is alsoreferred to as 3D scene generation.

Before using the system 100 in training or inference mode to track theinventory items, two types of camera calibrations: internal andexternal, are performed. In internal calibration, the internalparameters of the cameras 114 are calibrated. Examples of internalcamera parameters include focal length, principal point, skew, fisheyecoefficients, etc. A variety of techniques for internal cameracalibration can be used. One such technique is presented by Zhang in “Aflexible new technique for camera calibration” published in IEEETransactions on Pattern Analysis and Machine Intelligence, Volume 22,No. 11, November 2000.

In external calibration, the external camera parameters are calibratedin order to generate mapping parameters for translating the 2D imagedata into 3D coordinates in real space. In one embodiment, one subject(also referred to as a multi- joint subject), such as a person, isintroduced into the real space. The subject moves through the real spaceon a path that passes through the field of view of each of the cameras114. At any given point in the real space, the subject is present in thefields of view of at least two cameras forming a 3D scene. The twocameras, however, have a different view of the same 3D scene in theirrespective two-dimensional (2D) image planes. A feature in the 3D scenesuch as a left-wrist of the subject is viewed by two cameras atdifferent positions in their respective 2D image planes.

A point correspondence is established between every pair of cameras withoverlapping fields of view for a given scene. Since each camera has adifferent view of the same 3D scene, a point correspondence is two pixellocations (one location from each camera with overlapping field of view)that represent the projection of the same point in the 3D scene. Manypoint correspondences are identified for each 3D scene using the resultsof the image recognition engines 112 a to 112 n for the purposes of theexternal calibration. The image recognition engines identify theposition of a joint as (x, y) coordinates, such as row and columnnumbers, of pixels in the 2D image space of respective cameras 114. Inone embodiment, a joint is one of 19 different types of joints of thesubject. As the subject moves through the fields of view of differentcameras, the tracking engine 110 receives (x, y) coordinates of each ofthe 19 different types of joints of the subject used for the calibrationfrom cameras 114 per image.

For example, consider an image from a camera A and an image from acamera B both taken at the same moment in time and with overlappingfields of view. There are pixels in an image from camera A thatcorrespond to pixels in a synchronized image from camera B. Considerthat there is a specific point of some object or surface in view of bothcamera A and camera B and that point is captured in a pixel of bothimage frames. In external camera calibration, a multitude of such pointsare identified and referred to as corresponding points. Since there isone subject in the field of view of camera A and camera B duringcalibration, key joints of this subject are identified, for example, thecenter of left wrist. If these key joints are visible in image framesfrom both camera A and camera B then it is assumed that these representcorresponding points. This process is repeated for many image frames tobuild up a large collection of corresponding points for all pairs ofcameras with overlapping fields of view. In one embodiment, images arestreamed off of all cameras at a rate of 30 FPS (frames per second) ormore and a resolution of 720 pixels in full RGB (red, green, and blue)color. These images are in the form of one-dimensional arrays (alsoreferred to as flat arrays).

The large number of images collected above for a subject is used todetermine corresponding points between cameras with overlapping fieldsof view. Consider two cameras A and B with overlapping field of view.The plane passing through camera centers of cameras A and B and thejoint location (also referred to as feature point) in the 3D scene iscalled the “epipolar plane”. The intersection of the epipolar plane withthe 2D image planes of the cameras A and B defines the “epipolar line”.Given these corresponding points, a transformation is determined thatcan accurately map a corresponding point from camera A to an epipolarline in camera B's field of view that is guaranteed to intersect thecorresponding point in the image frame of camera B. Using the imageframes collected above for a subject, the transformation is generated.It is known in the art that this transformation is non-linear. Thegeneral form is furthermore known to require compensation for the radialdistortion of each camera's lens, as well as the non-linear coordinatetransformation moving to and from the projected space. In externalcamera calibration, an approximation to the ideal non-lineartransformation is determined by solving a non-linear optimizationproblem. This non-linear optimization function is used by the subjecttracking engine 110 to identify the same joints in outputs (arrays ofjoint data structures) of different image recognition engines 112 a to112 n, processing images of cameras 114 with overlapping fields of view.The results of the internal and external camera calibration are storedin a calibration database.

A variety of techniques for determining the relative positions of thepoints in images of cameras 114 in the real space can be used. Forexample, Longuet-Higgins published, “A computer algorithm forreconstructing a scene from two projections” in Nature, Volume 293, 10Sep. 1981. This paper presents computing a three- dimensional structureof a scene from a correlated pair of perspective projections whenspatial relationship between the two projections is unknown.Longuet-Higgins paper presents a technique to determine the position ofeach camera in the real space with respect to other cameras.Additionally, their technique allows triangulation of a subject in thereal space, identifying the value of the z-coordinate (height from thefloor) using images from cameras 114 with overlapping fields of view. Anarbitrary point in the real space, for example, the end of a shelf unitin one corner of the real space, is designated as a (0, 0, 0) point onthe (x, y, z) coordinate system of the real space.

In an embodiment of the technology, the parameters of the externalcalibration are stored in two data structures. The first data structurestores intrinsic parameters. The intrinsic parameters represent aprojective transformation from the 3D coordinates into 2D imagecoordinates. The first data structure contains intrinsic parameters percamera as shown below. The data values are all numeric floating pointnumbers. This data structure stores a 3×3 intrinsic matrix, representedas “K” and distortion coefficients. The distortion coefficients includesix radial distortion coefficients and two tangential distortioncoefficients. Radial distortion occurs when light rays bend more nearthe edges of a lens than they do at its optical center. Tangentialdistortion occurs when the lens and the image plane are not parallel.The following data structure shows values for the first camera only.Similar data is stored for all the cameras 114.

{  1: {   K: [[x, x, x], [x, x, x], [x, x, x]],  distortion_coefficients: [x, x, x, x, x, x, x, x]  }, }

The second data structure stores per pair of cameras: a 3×3 fundamentalmatrix (F), a 3×3 essential matrix (E), a 3×4 projection matrix (P), a3×3 rotation matrix (R) and a 3×1 translation vector (t). This data isused to convert points in one camera's reference frame to anothercamera's reference frame. For each pair of cameras, eight homographycoefficients are also stored to map the plane of the floor 220 from onecamera to another. A fundamental matrix is a relationship between twoimages of the same scene that constrains where the projection of pointsfrom the scene can occur in both images. Essential matrix is also arelationship between two images of the same scene with the conditionthat the cameras are calibrated. The projection matrix gives a vectorspace projection from 3D real space to a subspace. The rotation matrixis used to perform a rotation in Euclidean space. Translation vector “t”represents a geometric transformation that moves every point of a figureor a space by the same distance in a given direction. Thehomography_floor_coefficients are used to combine images of features ofsubjects on the floor 220 viewed by cameras with overlapping fields ofviews. The second data structure is shown below. Similar data is storedfor all pairs of cameras. As indicated previously, the x's representsnumeric floating point numbers.

{  1: {   2: {    F: [[x, x, x], [x, x, x], [x, x, x]],    E: [[x, x,x], [x, x, x], [x, x, x]],    P: [[x, x, x, x], [x, x, x, x], [x, x, x,x]],    R: [[x, x, x], [x, x, x], [x, x, x]],    t: [x, x, x],   homography_floor_coefficients: [x, x, x, x, x, x, x, x]   }  },  . .. . . . . }

Two Dimensional and Three Dimensional Maps

An inventory location, such as a shelf, in a shopping store can beidentified by a unique identifier in a map database (e.g., shelf_id).Similarly, a shopping store can also be identified by a uniqueidentifier (e.g., store_id) in a map database. The two dimensional (2D)and three dimensional (3D) maps database 140 identifies inventorylocations in the area of real space along the respective coordinates.For example, in a 2D map, the locations in the maps define twodimensional regions on the plane formed perpendicular to the floor 220i.e., XZ plane as shown in FIG. 2B. The map defines an area forinventory locations where inventory items are positioned. In FIG. 3, a2D location of the shelf unit shows an area formed by four coordinatepositons (x1, y1), (x1, y2), (x2, y2), and (x2, y1). These coordinatepositions define a 2D region on the floor 220 where the shelf islocated. Similar 2D areas are defined for all inventory displaystructure locations, entrances, exits, and designated unmonitoredlocations in the shopping store. This information is stored in the mapsdatabase 140.

In a 3D map, the locations in the map define three dimensional regionsin the 3D real space defined by X, Y, and Z coordinates. The map definesa volume for inventory locations where inventory items are positioned.In FIG. 2B, a 3D view 250 of shelf 1 in the shelf unit shows a volumeformed by eight coordinate positions (x1, y1, z1), (x1, y1, z2), (x1,y2, z1), (x1, y2, z2), (x2, y1, z1), (x2, y1, z2), (x2, y2, z1), (x2,y2, z2) defining a 3D region in which inventory items are positioned onthe shelf 1. Similar 3D regions are defined for inventory locations inall shelf units in the shopping store and stored as a 3D map of the realspace (shopping store) in the maps database 140. The coordinatepositions along the three axes can be used to calculate length, depthand height of the inventory locations as shown in FIG. 2B.

In one embodiment, the map identifies a configuration of units of volumewhich correlate with portions of inventory locations on the inventorydisplay structures in the area of real space. Each portion is defined bystarting and ending positions along the three axes of the real space.Like 2D maps, the 3D maps can also store locations of all inventorydisplay structure locations, entrances, exits and designated unmonitoredlocations in the shopping store.

The items in a shopping store are arranged in some embodiments accordingto a planogram which identifies the inventory locations (such asshelves) on which a particular item is planned to be placed. Forexample, as shown in an illustration 250 in FIG. 2B, a left half portionof shelf 3 and shelf 4 are designated for an item (which is stocked inthe form of cans).

Joints Data Structure

The image recognition engines 112 a-112 n receive the sequences ofimages from cameras 114 and process images to generate correspondingarrays of joints data structures. The system includes processing logicthat uses the sequences of images produced by the plurality of camera totrack locations of a plurality of subjects (or customers in the shoppingstore) in the area of real space. In one embodiment, the imagerecognition engines 112 a-112 n identify one of the 19 possible jointsof a subject at each element of the image, usable to identify subjectsin the area who may be moving in the area of real space, standing andlooking at an inventory item, or taking and putting inventory items. Thepossible joints can be grouped in two categories: foot joints andnon-foot joints. The 19^(th) type of joint classification is for allnon-joint features of the subject (i.e. elements of the image notclassified as a joint). In other embodiments, the image recognitionengine may be configured to identify the locations of handsspecifically. Also, other techniques, such as a user check-in procedureor biometric identification processes, may be deployed for the purposesof identifying the subjects and linking the subjects with detectedlocations of their hands as they move throughout the store.

Foot Joints:  Ankle joint (left and right) Non-foot Joints:  Neck  Nose Eyes (left and right)  Ears (left and right)  Shoulders (left andright)  Elbows (left and right)  Wrists (left and right)  Hip (left andright)  Knees (left and right) Not a joint

An array of joints data structures for a particular image classifieselements of the particular image by joint type, time of the particularimage, and the coordinates of the elements in the particular image. Inone embodiment, the image recognition engines 112 a-112 n areconvolutional neural networks (CNN), the joint type is one of the 19types of joints of the subjects, the time of the particular image is thetimestamp of the image generated by the source camera 114 for theparticular image, and the coordinates (x, y) identify the position ofthe element on a 2D image plane.

The output of the CNN is a matrix of confidence arrays for each imageper camera. The matrix of confidence arrays is transformed into an arrayof joints data structures. A joints data structure 310 as shown in FIG.3A is used to store the information of each joint. The joints datastructure 310 identifies x and y positions of the element in theparticular image in the 2D image space of the camera from which theimage is received. A joint number identifies the type of jointidentified. For example, in one embodiment, the values range from 1 to19. A value of 1 indicates that the joint is a left ankle, a value of 2indicates the joint is a right ankle and so on. The type of joint isselected using the confidence array for that element in the outputmatrix of CNN. For example, in one embodiment, if the valuecorresponding to the left-ankle joint is highest in the confidence arrayfor that image element, then the value of the joint number is “1”.

A confidence number indicates the degree of confidence of the CNN inpredicting that joint. If the value of confidence number is high, itmeans the CNN is confident in its prediction. An integer-Id is assignedto the joints data structure to uniquely identify it. Following theabove mapping, the output matrix of confidence arrays per image isconverted into an array of joints data structures for each image. In oneembodiment, the joints analysis includes performing a combination ofk-nearest neighbors, mixture of Gaussians, and various image morphologytransformations on each input image. The result comprises arrays ofjoints data structures which can be stored in the form of a bit mask ina ring buffer that maps image numbers to bit masks at each moment intime.

Subject Tracking Engine

The tracking engine 110 is configured to receive arrays of joints datastructures generated by the image recognition engines 112 a-112 ncorresponding to images in sequences of images from cameras havingoverlapping fields of view. The arrays of joints data structures perimage are sent by image recognition engines 112 a- 112 n to the trackingengine 110 via the network(s) 181. The tracking engine 110 translatesthe coordinates of the elements in the arrays of joints data structuresfrom 2D image space corresponding to images in different sequences intocandidate joints having coordinates in the 3D real space. A location inthe real space is covered by the field of views of two or more cameras.The tracking engine 110 comprises logic to determine sets of candidatejoints having coordinates in real space (constellations of joints) aslocated subjects in the real space. In one embodiment, the trackingengine 110 accumulates arrays of joints data structures from the imagerecognition engines for all the cameras at a given moment in time andstores this information as a dictionary in a subject database, to beused for identifying a constellation of candidate joints correspondingto located subjects. The dictionary can be arranged in the form ofkey-value pairs, where keys are camera ids and values are arrays ofjoints data structures from the camera. In such an embodiment, thisdictionary is used in heuristics-based analysis to determine candidatejoints and for assignment of joints to located subjects. In such anembodiment, a high- level input, processing and output of the trackingengine 110 is illustrated in table 1. Details of the logic applied bythe subject tracking engine 110 to create subjects by combiningcandidate joints and track movement of subjects in the area of realspace are presented in U.S. patent application Ser. No. 15/847,796,filed 19 Dec. 2017, now U.S. Pat. No. 10,055,853, issued on 21 Aug.2018, titled, “Subject Identification and Tracking Using ImageRecognition Engine” which is incorporated herein by reference.

TABLE 1 Inputs, processing and outputs from subject tracking engine 110in an example embodiment. Inputs Processing Output Arrays of joints dataCreate joints dictionary List of located subjects structures per imageReproject joint positions located in the real space and for each jointsin the fields of view of at a moment in time data structure cameras withcorresponding to an Unique ID overlapping fields of identificationinterval Confidence number view to candidate joints Joint number 2D (x,y) position in image space

Subject Data Structure

The subject tracking engine 110 uses heuristics to connect jointsidentified by the image recognition engines 112 a-112 n to locatesubjects in the area of real space. In doing so, the subject trackingengine 110, at each identification interval, creates new locatedsubjects for tracking in the area of real space and updates thelocations of existing tracked subjects matched to located subjects byupdating their respective joint locations. The subject tracking engine110 can use triangulation techniques to project the locations of jointsfrom 2D image space coordinates (x, y) to 3D real space coordinates (x,y, z). FIG. 3B shows the subject data structure 320 used to store thesubject. The subject data structure 320 stores the subject related dataas a key-value dictionary. The key is a “frame_id” and the value isanother key-value dictionary where key is the camera_id and value is alist of 18 joints (of the subject) with their locations in the realspace. The subject data is stored in the subject database 150. A subjectis assigned a unique identifier that is used to access the subject'sdata in the subject database.

In one embodiment, the system identifies joints of a subject and createsa skeleton (or constellation) of the subject. The skeleton is projectedinto the real space indicating the position and orientation of thesubject in the real space. This is also referred to as “pose estimation”in the field of machine vision. In one embodiment, the system displaysorientations and positions of subjects in the real space on a graphicaluser interface (GUI). In one embodiment, the subject identification andimage analysis are anonymous, i.e., a unique identifier assigned to asubject created through joints analysis does not identify personalidentification information of the subject as described above.

For this embodiment, the joints constellation of a subject, produced bytime sequence analysis of the joints data structures, can be used tolocate the hand of the subject. For example, the location of a wristjoint alone, or a location based on a projection of a combination of awrist joint with an elbow joint, can be used to identify the location ofhand of a subject.

Subject Persistence Analysis

The joints analysis performed by the subject tracking engine 110 in anidentification interval identifies constellation of joints. Theidentification interval can correspond to one image capture cycle or caninclude multiple image capture cycles. The constellation of jointslocated in an identification interval can belong to new subjects whohave entered the area of real space in the current identificationinterval or can represent updated locations of the previously trackedsubjects in earlier identification intervals. Sometimes, a subjectlocated and tracked in an earlier identification interval can be missingin an intermediate identification interval before the currentidentification interval. This can happen for a variety of reasons,including, due to the subject moving to a designated unmonitoredlocation in the area of real space or due to an error in subjecttracking.

When a located subject is identified in the current identificationinterval, the technology disclosed performs the subject persistenceanalysis before tracking the located subject as a new subject andassigning it a new unique identifier. The system matches locatedsubjects from the current identification interval with tracked subjectsfrom an immediately preceding identification interval. Located subjectsthat are matech with tracked subjects can be tagged as the matchingtracked subject. Located subjects that are not matched with trackedsubjects are subjected to additional processing. For example, the systemdetermines if a tracked subject in one or more earlier identificationintervals is missing (i.e. not matched to a located subject) in theimmediately preceding identification interval. Such a missing trackedsubject can be evaluated as a potential match for the unmatched locatedsubject (candidate subject) in the current identification interval.

For example, the system can include logic that processes the set oftracked subjects in the subject database 150 to detect a missing trackedsubject present in the database. The missing tracked subject is notlocated in a first preceding identification interval but is tracked in asecond preceding identification interval. The first precedingidentification interval follows the second preceding identificationinterval. The system includes logic to locate a candidate locatedsubject located from the current identification interval which followsthe first preceding identification interval. The current identificationinterval can also be referred to as the third identification interval.

The technology disclosed matches the candidate located subject locatedfrom the current identification interval to the missing tracked subjectlocated in the second preceding identification interval. If the missingtracked subject matches the candidate located subject, the missingtracked subject is updated in the database using the location of thecandidate subject, and marked as no longer missing. This enablespersistence of subjects in the area of real space even when a trackedsubject is missed and not located in an identification interval.

It is understood that variations of subject persistence analysis arepossible. For example, the system can match newly located candidatesubject in the current identification interval to a missing trackedsubject who is not located and tracked by the system for more than oneintermediate identification intervals before the current identificationinterval.

The following sections present three example scenarios in which subjectpersistence analysis can be performed in an area of real space.

Single New Subject Persistence Analysis

The first example includes performing subject persistence over threeidentification intervals to match a missing tracked subject located inthe second preceding identification interval to a candidate locatedsubject located from the current (or third) identification interval. Thesystem detects a condition in which a number of located subjects in thecurrent set does not match the number of located subjects from a firstpreceding identification interval in the plurality of previousintervals. Upon detection of the condition, the system compares at leastone of the located subjects in the current set with the set of locatedsubjects from a second preceding identification interval in theplurality of previous identification intervals, that precedes the firstpreceding identification interval. The following example uses threeidentification intervals to illustrate this scenario. However, theprocess can be applied to more than three identification intervals.

FIG. 4A presents a side view 402 of an area of real space in which threesubjects 440, 442 and 444 are tracked in a second precedingidentification interval at time t_0. The subjects are stored in thedatabase with their respective unique tracking identifiers and locationinformation. The positions of the three subjects in a top view 404(looking down from roof) is also shown in a top view 404. As describedabove, the positions of the subjects in the area of real space isidentified by their respective subject data structures 320. The subjectdata structures include locations of joints in three dimensions (x, y,z) of the area of real space. In another embodiment, the positions ofthe joints or other feature of the subjects are represented in the twodimensional (abbreviated 2D) image space (x, y). The subject 442 who istracked in the second preceding identification interval is missing in afirst preceding identification interval at time t_1 as shown in FIG. 4B.Both side view (402) and top view (404) show subjects 440 and 444tracked in the first preceding identification intervals. A candidatesubject 442A is located in a current identification interval at time t_2as shown FIG. 4C. The candidate located subject is visible in the sideview 402 and the top view 404.

The technology disclosed performs the subject persistence analysis todetermine if the candidate located subject 442A is a new subject whoentered the area of real space during the current identificationinterval or if the candidate located subject 442A is the missing trackedsubject 442 who was tracked in the second preceding identificationinterval but is missing in the first preceding identification interval.

FIG. 5 presents a flowchart with example process steps to perform thesubject persistence for one candidate located subject located from thecurrent identification interval. The process starts at step 502. Thesystem locates subjects in the current identification cycle at time t_2.In one embodiment, the system uses joints analysis as described above tolocate subjects as constellation of joints. In another embodiment, thesystem can use other features of the subjects such as facial featuresindependently or in combination with joints to locate subjects in thearea of real space.

At step 506, the process matches the subjects located in the currentidentification interval at t_2 to tracked subjects located in the firstpreceding identification interval at time t_1. In one embodiment, theprocess uses the logic applied by the subject tracking engine 110 tocreate subjects by combining candidate joints and track movement ofsubjects in the area of real space as presented in U.S. patentapplication Ser. No. 15/847,796, filed 19 Dec. 2017, now U.S. Pat. No.10,055,853, issued on 21 Aug. 2018, titled, “Subject Identification andTracking Using Image Recognition Engine” which is incorporated herein byreference. At step 508, the system determines if all subjects located inthe current identification interval match to the tracked subjects in thefirst preceding identification interval. If all subjects match then thesystem repeats the steps 504 and 506 for a next identification interval.In one embodiment, in this step, the system detect a condition in whicha number of located subjects in the current set does not match thenumber of located subjects from a first preceding identificationinterval in the plurality of previous intervals. If the condition istrue (step 508), then the system compares at least one of the locatedsubjects in the current set with the set of tracked subjects from asecond preceding identification interval in the plurality of previousidentification intervals, that precedes the first precedingidentification interval.

If a candidate located subject located from the current identificationinterval does not match to any tracked subject in the first precedingidentification interval, the technology disclosed determines if there isa missing tracked subject who was located and tracked in the secondpreceding identification interval but was missing in the first precedingidentification interval following the second preceding identificationinterval. If the system identifies a missing tracked subject who istracked in the second preceding identification interval but is missingin the first preceding identification interval, the process continues ata step 516. Otherwise, if the system does not identify a missing trackedsubject in the second preceding identification interval, the systemstarts tracking the candidate located subject located from the currentidentification interval by assigning this subject a new trackingidentifier. This is the case when all tracked subjects in the firstpreceding identification interval match corresponding tracked subjectsin the second preceding identification interval.

In the example presented in FIGS. 4A to 4C, the subject 442A (shown inFIG. 4C) is the candidate located subject located from the currentidentification interval and the subject 442 (shown in FIG. 4A) is themissing tracked subject. If the system determines that there is nomissing tracked subject at step 512, the candidate located subject 442Ais assigned a unique tracking identifier and the system starts trackingthe subject during the current identification interval.

The process to match the missing tracked subject and the candidatelocated subject is described in the following steps of the process flow.In the process step 516, the system applies a time constraint heuristicbefore matching the location of the candidate located subject locatedfrom the current identification interval to the location of the missingtracked subject in the second preceding identification interval. Thesystem calculates for example a difference in a timestamp of location ofthe candidate located subject and a timestamp of location of the missingtracked subject. The timestamps can be identifiers of the identificationintervals, or can be specific timestamps within an identificationinterval that includes a plurality of image capture cycles. Thetimestamp, for example, can be represented as t_2 for the candidatesubject located from the current identification interval, and t_0 forthe missing tracked subject located in the second precedingidentification interval. If an identification interval matches an imagecapture cycle of the cameras, the timestamps can match the time at whichthe images are captured in the image capture cycles. The differencebetween the timestamps i.e., t_2-t_0 is compared with a timingthreshold. In one example, the timing threshold is 10 seconds. It isunderstood that other values of timing threshold greater or less than 10seconds can be used. The timestamps of detection of joints of thesubjects at image capture cycles can also be used for calculation ofthis time difference. If the difference in timestamps is less than thetiming threshold then the system matches locations of the candidatelocated subject and the missing tracked subject.

The system calculates a distance between a location of the candidatelocated subject (p_2) and a location of the missing tracked subject(p_0) in the area of real space, i.e., (p_2-p_1). In one embodimentusing joints analysis as described above, the distance is calculatedusing locations of joints in the constellations of joints of thecandidate located subject and the missing tracked subject. The distancecan be calculated as a Euclidean distance between two pointsrepresenting the corresponding joints in the respective constellationsof joints. The Euclidean distance can be calculated both in the 3D realspace and the 2D image space. The Euclidean distance represents thedistance the subject has moved from an initial position in the secondpreceding identification interval to a new position in the currentidentification interval. This distance is then compared with a distancethreshold. If the distance is less than the distance threshold then thecandidate located subject is matched to the missing tracked subject. Anexample of the distance threshold distance is 1 meter. Other values forthe distance threshold, greater than 1 meter or less than 1 meter can beused. If the difference between the timestamps of the location of thecandidate located subject and the missing tracked subject is greaterthan the timing threshold or the distance between the candidate locatedsubject and the missing tracked subject is greater than the distancethreshold (step 518), the system start does not match the candidatesubject to the missing tracked subject, and can identify it as a newlocated subject at step 514. Otherwise, the process to link thecandidate located subject and the missing tracked subject continues atstep 520.

At step 520, before linking the candidate located subject located fromthe current identification interval to the missing tracked subjectlocated in the second preceding identification interval, the systemapplies “sink” constraints. Sink constraints can include calculatingdistances between locations of the candidate located subject and of themissing tracked subject to locations in the area of real space that canprovide sources and sinks of subjects, such as entrances or exits fromthe area of the real space. In one embodiment, the distance calculationuses a boundary location of the entrance or exit region. The distance ofthe candidate located subject to the location i.e., d(p_2-sink) and thedistance of the missing tracked subject to the location i.e.,d(p_0-sink) are compared with a distance threshold. If either of thesedistances is less than the distance threshold, (step 522) and the systemcan start tracking candidate located subject at step 514. An example ofdistance threshold at step 520 is 1 meter, in other embodiments,distance threshold values greater than 1 meter or less than 1 meter canbe used. In one embodiment, the threshold depends on the length (or timeduration) of the identification intervals and the distance a subject canmove in that time duration. If the sink constraints are satisfied, i.e.,both the candidate located subject and the missing tracked subject arepositioned farther from entrance and exits by more than the distancethreshold, the system can update the missing tracked subject in thedatabase using the candidate located subject located from the currentidentification interval (step 524). The process ends at step 526.

Multiple New Subjects Persistence Analysis

The second example scenario for performing subject persistence analysisusing the technology disclosed is presented in FIGS. 6A and 6B. Thisexample considers the scenario in which the set of tracked subjects froma first preceding identification interval includes N members, and theset of located subjects from the current identification intervalincludes N members plus one or more candidate located subjects. Thesystem can employ logic to make the process of linking N members of theset of located subjects from the current identification interval to Nmembers of the set of tracked subjects from the first precedingidentification interval, more efficient. This improvement in efficiencycan be achieved by prioritizing members of the set of located subjectsfrom the current identification interval to identify a set of N locatedsubjects to link to the set of N tracked subjects from the firstpreceding identification interval using relative locations of thelocated subjects. In one embodiment, the prioritization of the membersof the set of located subjects from the current identification intervalto identify the set of N located subjects can include calculatingdistance between pairs of located subjects from the currentidentification interval. The system then identifies the set of N locatedsubjects by comparing the calculated distance with a distance secondthreshold such as 1 meter. Located members satisfying the distancethreshold can be evaluated for matching with tracked member for thepreceding identification interval with higher priority than those thatdo not meet the distance threshold.

The example presented in FIGS. 6A and 6B illustrates this scenario bytracking three subjects 640, 642 and 644 in the first precedingidentification interval at t_1 as shown in FIG. 6A. The three trackedsubjects 640, 642, and 644 are stored in the subject database 150 withtheir unique tracking identifiers. Five subjects 640A, 642A, 644A, 646and 648 are located in the current identification interval at t_2 asshown in FIG. 6B. The set of subjects located in the currentidentification interval has more than one member subjects not tracked inthe first preceding identification interval. The system compares the setof tracked subjects present in the database that are tracked inpreceding identification intervals to detect the condition that morethan one subjects not tracked in preceding identification intervals arelocated in the current identification interval.

The system prioritizes the set of subjects (N plus more than onecandidate located subjects) located from the current identificationinterval to determine a set S of located subjects in the currentidentification interval. In one embodiment, the cardinality of the set Sis equal to the cardinality of the set N of tracked subjects inpreceding identification interval. In other embodiment, the cardinalityof the set S can be less than the set N of tracked subjects in precedingidentification intervals. In one embodiment, the membership of set S isdetermined such that the three dimensional or two dimensional Euclideandistance between any two members in the set S is less than a distancethreshold. An example of the distance threshold is 1 meter. In FIG. 6B,a circle 610 identifies the set S of located subjects in the currentidentification interval which includes subjects 640A, 642A, and 644A. Inthis example, the cardinality of the set S equals the cardinality of theset N of tracked subjects in the preceding identification interval.

The system includes logic that matches a member of the set S of locatedsubjects in the current identification interval to members of the set Nof tracked subjects in the first preceding identification interval. If amember of the set S matches a member of the set N of tracked subjects,the system links the matched located subjects to the matched trackedsubjects and updates the tracked subject in the database using thematched located subject. In one embodiment, members of the set S oflocated subjects are matched to members of the set N of tracked subjectsin the first preceding identification interval using positions of jointsin their respective constellation of joints. The distance betweenpositions of joints of members of the set S of located subjects and theset N of tracked subjects are compared with the second threshold (e.g.,1 meter). If there is a unique match for each member of the set S oflocated subjects in the current identification interval to a trackedsubject in the first preceding identification interval then the systemupdates the tracked subject in the database using the matched member ofthe set S of located subjects. Before linking the tracked subject to thelocated subject in the current identification interval, the system canapply sink constraints as discussed above to the matched located andtracked subjects to determine that they are away from the exits andentrances to the area of real space by more than a third thresholddistance.

FIG. 7 presents a process flowchart to perform subject persistence inthe above described scenario. The process starts at a step 702. Thesystem locates subjects in the area of real space in the currentidentification interval at a step 704. The number of located subjects inthe current identification interval is compared to the number of trackedsubjects in the first preceding identification interval at step 706. Forexample, consider the example illustration in FIG. 6B indicating fivesubjects located in the current identification interval. Suppose therewere three subjects tracked in the first preceding identificationinterval. By comparing the number of located subjects in the currentidentification interval to the number of tracked subjects in the firstpreceding identification interval, the system determines that more thanone candidate subjects are located in the current identificationinterval (step 708). In one embodiment, the system compares the numberof located subjects in the current identification interval to the numberof tracked subjects in preceding identification interval to determinethat more than one candidate subjects are located in the currentidentification interval at the step 706. In other words, the number oflocated subjects in the current identification interval is more than oneplus the number of tracked subjects in the preceding identificationinterval. If there is only one additional member in the currentidentification interval, then the technique presented above in FIG. 5can be applied. If there is only one additional member in the currentidentification interval and that member is positioned close to adesignated unmonitored location in the area of real space (such as arestroom) then the technique presented below in FIG. 9 can be used.

The system identifies a set S of located subjects in the currentidentification interval (step 710) as explained in the followingexample. Consider M_2 subjects are located in the current identificationinterval at time t_2, whereas the subjects M_2 are indexed as 0, 1, 2, .. . . , M_2-1 and M_0 subjects are tracked in the first precedingidentification interval at time t_1, indexed as 0, 1, 2, . . . . ,M_0-1. Further, suppose that locations of the located subjects in thecurrent identification interval are represented as p_{2, i} for i=0, 1,2, . . . . , M_2-1 and locations of the tracked subjects in the firstpreceding identification interval are represented as q_{0, i} for i=0,1, 2, . . . . , M_0-1. At the step 710, a set S of located subjects inthe current identification interval is determined such that for any twosubjects p_{2, i} and p_{2, j} the distance d (p_{2, i}, p_{2, j }) isless than a second threshold, e.g., 1 meter. The distance can becalculated in the 3D real space or 2D image space using locations ofjoints in the constellation of joints of respective subjects. It isunderstood that other values of the distance threshold greater than orless than 1 meter can be used.

The members of the set S of located subjects are then matched to trackedsubjects in the first preceding identification interval at step 712. Thelocation of each located subjects p_{2, i} member of the set S ismatched to locations of tracked subjects q_{0, j} in the first precedingidentification interval to determine the distance d (p_{2, i}, q_{0, j}). If the distance “d” is less than a second threshold, such as 1meter, and one member p_{2, i} in the set S of located subjects matchesto only one tracked subject q_{0, j } using the above threshold, thenthe system determines that there is a match between the located subjectand tracked subject located in the preceding identification interval(step 714).

If a member of the set S of located subjects does not match to a trackedsubject in the above process step, the located subject can be assigned anew tracking identifier at a step 716. The system can then starttracking the located subject in the current identification interval. Thesubject is stored in the subject database with a unique trackingidentifier.

When a member of the set S of located subjects in the currentidentification interval is matched to a tracked subject in the firstpreceding identification interval, the system determines that no othermember of the set S of located subjects matches that tracked subject.For a member p_{2, i} of the set S of located subjects that uniquelymatches to a tracked subject q_{0, j }, the sink constraints are appliedat a step 718. The sink constraints determine if the member of the set Sof located subjects or the tracked subject are closer to an entrance toor exit from the area of real space by a third threshold distance asdescribed in the step 520 in the flowchart in FIG. 5. If the sinkconstraint is satisfied (step 720) for the member of the set S of thelocated subjects and the tracked subject (i.e., both the member of theset S of located subjects and the tracked subject are farther from thesink than the third threshold), the tracked subject q_{0, j in the firstpreceding identification interval is updated in the database using themember p_{2, i} of the set S of located subjects (step 722). The processends at a step 724.

Subject Persistence Analysis for Designated Unmonitored Location

A third example scenario for performing subject persistence analysisusing the technology disclosed is presented in FIGS. 8A to 8C. Thisexample illustrates subject persistence when a subject moves to adesignated unmonitored location, such as a restroom, in the area of realspace. The subject is then not tracked in the following one or moreidentification intervals during which the subject is present in thedesignated unmonitored location. The system again locates the missingtracked subject during a following identification interval in which thesubject moves out of the designated unmonitored location and ispositioned in the field of view of one or more cameras 114.

FIG. 8A illustrates a top view (looking downwards) of an area of realspace that includes a designated unmonitored location 804 such as arestroom. The designated unmonitored location 804 is not in the field ofview of cameras 114. Subjects can enter or leave the designatedunmonitored location through a door 806. There are five subjects 840,842, 844, 846, and 848 in the set of tracked subjects in a secondpreceding identification interval at time t_0 as shown in FIG. 8A. In afirst preceding identification interval at time t_1, there are fourtracked subjects 840, 842, 844, and 846 in the set of tracked subjectsas shown in FIG. 8B. The tracked subject 848 in the second precedingidentification interval is missing in the first preceding identificationinterval. The location of the missing tracked subject 848 is close tothe designated unmonitored location in the second precedingidentification interval before the first preceding identificationinterval in which the subject 848 is missing.

FIG. 8C shows a candidate located subject 848A positioned near thedesignated unmonitored location 804 in a current identification intervalat time t_2 after the first preceding identification interval at timet_1. Before starting tracking the candidate located subject 848A in thecurrent identification interval, the technology disclosed performs thesubject persistence analysis to link the candidate located subject 848Ato the missing tracked subject 848. The missing tracked subject 848 waslocated in the second preceding identification interval but was notlocated in the first preceding identification interval following thesecond preceding identification interval. Before the candidate locatedsubject is matched to the missing tracked subject, the technologydisclosed can determine that no subject (other than the missing trackedsubject 848) was present close to the designated unmonitored location inthe second preceding identification interval and no other subject (otherthan the missing tracked subject 848) entered the designated unmonitoredlocation in the first preceding identification interval.

The system matches the locations of all tracked subjects in the secondpreceding identification interval to the location of designatedunmonitored location to determine that only the missing tracked subject848 is positioned close to the unmonitored location in the secondpreceding identification interval. In one embodiment, a distance iscalculated between locations of the tracked subjects in the secondpreceding identification interval and a point (in 3D real space or 2Dimage space) on the door 806. The system determines which trackedsubjects are close to the designated unmonitored location 804 bycomparing their respective distances to the designated unmonitoredlocation with a third threshold. An example value of the third thresholddistance is 1 meter. If missing tracked subject 848 is the only subjectclose to the door 806 in the second preceding identification intervaland the candidate subject 848A is the only candidate located subjectlocated from the current identification interval who is positioned closeto the designated unmonitored location then the system links the missingtracked subject 848 to the candidate located subject 848A. The systemupdated the missing tracked subject 848 in the database using thecandidate located subject 848A in the current identification intervaland continues tracking the subject 848 in the current identificationinterval.

FIG. 9 is a flowchart presenting process steps to link a candidatelocated subject located from the current identification interval to amissing tracked subject in the second preceding identification intervalif both the candidate located subject and the missing tracked subjectare positioned close to the designated unmonitored location inrespective identification intervals. The process starts at step 902. Theprocess steps 904, 906, 908, 910, and 912 perform similar operations asdescribed for process steps 504, 506, 508, 510, and 512 respectively. Ata step 916, the distances of the tracked subjects in the secondpreceding identification interval and the located subjects in thecurrent identification interval to a designated unmonitored location arecalculated. Suppose there are M_0 subjects in the set of trackedsubjects in the second preceding identification interval and the trackedsubjects are indexed as 0, 1, 2, . . . . , k-1. The locations of thetracked subjects are given as p_0, p_1, p_2, . . . . , p_{k-1},respectively. The system calculates distances of the tracked subjects tolocation of the designated unmonitored location as d(p_i, B) where B isthe location of the designated unmonitored location in the threedimensional real space or two dimensional image space.

The distances of the tracked subjects to the designated unmonitoredlocation are compared with a distance threshold such as 1 meter. If onlyone tracked subject in the second preceding identification interval ispositioned closer to the designated unmonitored location than the thirdthreshold, a similar distance calculation between locations of subjectslocated in the current identification interval and location of thedesignated unmonitored location is performed. If only one subjectlocated in the current identification interval is positioned closer tothe unmonitored designated location then the condition at step 918becomes true. Otherwise, the system can assign a new tracking identifierto the candidate located subject located from the current identificationinterval and starts tracking the subject (step 914).

As part of linking the missing tracked subject located in the secondpreceding identification interval to the candidate located subjectlocated from the current identification interval, additional constraintscan be checked at a step 920. It is determined that no other trackedsubjects from the second preceding identification interval and the firstpreceding identification intervals who were located closer to thedesignated unmonitored location than the distance threshold (other thanthe missing tracked subject at step 910) are missing in the currentidentification interval. This is to avoid incorrect matching of thecandidate located subject to the missing tracked subject. If only onetracking subject positioned close to the designated unmonitored locationin the second preceding identification interval is not tracked in thefirst preceding identification interval and only one candidate subjectis located close to the designated unmonitored location in the currentidentification interval, then the system checks the followingconstraint. The system determines that no other tracked subject enteredthe designated unmonitored location (step 922) by performing the processsteps 904 to 912. If no other tracked subject entered the designatedunmonitored location in the second preceding identification interval andthe first preceding identification interval then the system links themissing tracked subject located in the second preceding identificationinterval using the candidate located subject located from the currentidentification interval (step 924). The system then continues trackingthe missing tracked subject in the current identification interval usingthe location of the candidate located subject. The process ends at step926.

Network Configuration

FIG. 10 presents architecture of a network hosting the subjectpersistence processing engine 180 which is hosted on the network node104. The system includes a plurality of network nodes 101 a, 101 b, 101n, and 102 in the illustrated embodiment. In such an embodiment, thenetwork nodes are also referred to as processing platforms. Processingplatforms (network nodes) 103, 101 a-101 n, and 102 and cameras 1012,1014, 1016, . . . , 1018 are connected to network(s) 1081. A similarnetwork hosts the matching engine 170 which is hosted on the networknode 103.

FIG. 10 shows a plurality of cameras 1012, 1014, 1016, . . . , 1018connected to the network(s). A large number of cameras can be deployedin particular systems. In one embodiment, the cameras 1012 to 1018 areconnected to the network(s) 1081 using Ethernet-based connectors 1022,1024, 1026, and 1028, respectively. In such an embodiment, theEthernet-based connectors have a data transfer speed of 1 gigabit persecond, also referred to as Gigabit Ethernet. It is understood that inother embodiments, cameras 114 are connected to the network using othertypes of network connections which can have a faster or slower datatransfer rate than Gigabit Ethernet. Also, in alternative embodiments, aset of cameras can be connected directly to each processing platform,and the processing platforms can be coupled to a network.

Storage subsystem 1030 stores the basic programming and data constructsthat provide the functionality of certain embodiments of the presentinvention. For example, the various modules implementing thefunctionality of the subject persistence processing engine 180 may bestored in storage subsystem 1030. The storage subsystem 1030 is anexample of a computer readable memory comprising a non-transitory datastorage medium, having computer instructions stored in the memoryexecutable by a computer to perform all or any combination of the dataprocessing and image processing functions described herein includinglogic to identify match located subjects with tracked subject, logic tolink subjects in an area of real space with a user account, to determinelocations of tracked subjects represented in the images, match thetracked subjects with user accounts by identifying locations of mobilecomputing devices executing client applications in the area of realspace by processes as described herein. In other examples, the computerinstructions can be stored in other types of memory, including portablememory, that comprise a non-transitory data storage medium or media,readable by a computer.

These software modules are generally executed by a processor subsystem1050. A host memory subsystem 1032 typically includes a number ofmemories including a main random access memory (RAM) 1134 for storage ofinstructions and data during program execution and a read-only memory(ROM) 1036 in which fixed instructions are stored. In one embodiment,the RAM 1034 is used as a buffer for storing point cloud data structuretuples generated by the inventory event location processing engine 180.

A file storage subsystem 1040 provides persistent storage for programand data files. In an example embodiment, the storage subsystem 1040includes four 120 Gigabyte (GB) solid state disks (SSD) in a RAID 0(redundant array of independent disks) arrangement identified by anumeral 1042. In the example embodiment, maps data in the maps database140, inventory events data in the inventory events database 150,inventory item activity data in the inventory item activity database160, and the inventory item correlation data in the inventory itemcorrelation database 170 which is not in RAM is stored in RAID 0. In theexample embodiment, the hard disk drive (HDD) 1046 is slower in accessspeed than the RAID 0 1042 storage. The solid state disk (SSD) 1044contains the operating system and related files for the inventory eventlocation processing engine 180.

In an example configuration, four cameras 1012, 1014, 1016, 1018, areconnected to the processing platform (network node) 103. Each camera hasa dedicated graphics processing unit GPU 1 1062, GPU 2 1064, GPU 3 1066,and GPU 4 1068, to process images sent by the camera. It is understoodthat fewer than or more than three cameras can be connected perprocessing platform. Accordingly, fewer or more GPUs are configured inthe network node so that each camera has a dedicated GPU for processingthe image frames received from the camera. The processor subsystem 1050,the storage subsystem 1030 and the GPUs 1062, 1064, and 1066 communicateusing the bus subsystem 1054.

A network interface subsystem 1070 is connected to the bus subsystem1054 forming part of the processing platform (network node) 104. Networkinterface subsystem 1070 provides an interface to outside networks,including an interface to corresponding interface devices in othercomputer systems. The network interface subsystem 1070 allows theprocessing platform to communicate over the network either by usingcables (or wires) or wirelessly. The wireless radio signals 1075 emittedby the mobile computing devices 120 in the area of real space arereceived (via the wireless access points) by the network interfacesubsystem 1170 for processing by the matching engine 170. A number ofperipheral devices such as user interface output devices and userinterface input devices are also connected to the bus subsystem 1154forming part of the processing platform (network node) 104. Thesesubsystems and devices are intentionally not shown in FIG. 10 to improvethe clarity of the description. Although bus subsystem 1054 is shownschematically as a single bus, alternative embodiments of the bussubsystem may use multiple busses.

In one embodiment, the cameras 114 can be implemented using Chameleon31.3 MP Color USB3 Vision (Sony ICX445), having a resolution of 1288×964,a frame rate of 30 FPS, and at 1.3 MegaPixels per image, with VarifocalLens having a working distance (mm) of 300-∞, a field of view field ofview with a ⅓″ sensor of 98.2°-23.8°.

Subject Identification Analysis

The following description provides examples of algorithms foridentifying tracked subjects by matching them to their respective useraccounts. As described above, the technology disclosed links locatedsubjects in the current identification interval to tracked subjects inpreceding identification intervals by performing subject persistenceanalysis. In the case of a cashier-less store the subjects move in theaisles and open spaces of the store and take items from shelves. Thetechnology disclosed associates the items taken by tracked subjects totheir respective shopping cart or log data structures. The technologydisclosed uses one of the following check-in techniques to identifytracked subjects and match them to their respective user accounts. Theuser accounts have information such as preferred payment method for theidentified subject. The technology disclosed can automatically chargethe preferred payment method in the user account in response toidentified subject leaving the shopping store. In one embodiment, thetechnology disclosed compares located subjects in current identificationinterval to tracked subjects in previous identification intervals inaddition to comparing located subjects in current identificationinterval to identified (or checked in) subjects (linked to useraccounts) in previous identification intervals. In another embodiment,the technology disclosed compares located subjects in currentidentification interval to tracked subjects in previous intervals inalternative to comparing located subjects in current identificationinterval to identified (or tracked and checked-in) subjects (linked touser accounts) in previous identification intervals.

In a shopping store, the shelves and other inventory display structurescan be arranged in a variety of manners, such as along the walls of theshopping store, or in rows forming aisles or a combination of the twoarrangements. FIG. 11 shows an arrangement of shelves, forming an aisle116 a, viewed from one end of the aisle 116 a. Two cameras, camera A 206and camera B 208 are positioned over the aisle 116 a at a predetermineddistance from a roof 230 and a floor 220 of the shopping store above theinventory display structures, such as shelves. The cameras 114 comprisecameras disposed over and having fields of view encompassing respectiveparts of the inventory display structures and floor area in the realspace. The coordinates in real space of members of a set of candidatejoints, located as a subject, identify locations of the subject in thefloor area. In FIG. 11, the subject 240 is holding the mobile computingdevice 118 a and standing on the floor 220 in the aisle 116 a. Themobile computing device can send and receive signals through thewireless network(s) 181. In one example, the mobile computing devices120 communicate through a wireless network using for example a Wi- Fiprotocol, or other wireless protocols like Bluetooth, ultra-wideband,and ZigBee, through wireless access points (WAP) 250 and 252.

In the example embodiment of the shopping store, the real space caninclude all of the floor 220 in the shopping store from which inventorycan be accessed. Cameras 114 are placed and oriented such that areas ofthe floor 220 and shelves can be seen by at least two cameras. Thecameras 114 also cover at least part of the shelves 202 and 204 andfloor space in front of the shelves 202 and 204. Camera angles areselected to have both steep perspective, straight down, and angledperspectives that give more full body images of the customers. In oneexample embodiment, the cameras 114 are configured at an eight (8) footheight or higher throughout the shopping store.

In FIG. 11, the cameras 206 and 208 have overlapping fields of view,covering the space between a shelf A 202 and a shelf B 204 withoverlapping fields of view 216 and 218, respectively. A location in thereal space is represented as a (x, y, z) point of the real spacecoordinate system. “x” and “y” represent positions on a two- dimensional(2D) plane which can be the floor 220 of the shopping store. The value“z” is the height of the point above the 2D plane at floor 220 in oneconfiguration.

FIG. 12 illustrates the aisle 116 a viewed from the top of FIG. 11,further showing an example arrangement of the positions of cameras 206and 208 over the aisle 116 a. The cameras 206 and 208 are positionedcloser to opposite ends of the aisle 116 a. The camera A 206 ispositioned at a predetermined distance from the shelf A 202 and thecamera B 208 is positioned at a predetermined distance from the shelf B204. In another embodiment, in which more than two cameras arepositioned over an aisle, the cameras are positioned at equal distancesfrom each other. In such an embodiment, two cameras are positioned closeto the opposite ends and a third camera is positioned in the middle ofthe aisle. It is understood that a number of different cameraarrangements are possible.

Matching Engine

The matching engine 170 includes logic to identify tracked subjects bymatching them with their respective user accounts by identifyinglocations of mobile devices (carried by the tracked subjects) that areexecuting client applications in the area of real space. In oneembodiment, the matching engine uses multiple techniques, independentlyor in combination, to match the tracked subjects with the user accounts.The system can be implemented without maintaining biometric identifyinginformation about users, so that biometric information about accountholders is not exposed to security and privacy concerns raised bydistribution of such information.

In one embodiment, a customer (or a subject) logs in to the system usinga client application executing on a personal mobile computing deviceupon entering the shopping store, identifying an authentic user accountto be associated with the client application on the mobile device. Thesystem then sends a “semaphore” image selected from the set ofunassigned semaphore images in the image database 166 to the clientapplication executing on the mobile device. The semaphore image isunique to the client application in the shopping store as the same imageis not freed for use with another client application in the store untilthe system has matched the user account to a tracked subject. After thatmatching, the semaphore image becomes available for use again. Theclient application causes the mobile device to display the semaphoreimage, which display of the semaphore image is a signal emitted by themobile device to be detected by the system. The matching engine 170 usesthe image recognition engines 112 a-n or a separate image recognitionengine (not shown in FIG. 1) to recognize the semaphore image anddetermine the location of the mobile computing device displaying thesemaphore in the shopping store. The matching engine 170 matches thelocation of the mobile computing device to a location of a trackedsubject. The matching engine 170 then links the tracked subject (storedin the subject database 140) to the user account (stored in the useraccount database 164) linked to the client application for the durationin which the subject is present in the shopping store. No biometricidentifying information is used for identifying the subject by matchingthe tracking subject with the user account, and none is stored insupport of this process. That is, there is no information in thesequences of images used to compare with stored biometric informationfor the purposes of matching the tracked subjects with user accounts insupport of this process.

In other embodiments, the matching engine 170 uses other signals in thealternative or in combination from the mobile computing devices 120 tolink the tracked subjects to user accounts. Examples of such signalsinclude a service location signal identifying the position of the mobilecomputing device in the area of the real space, speed and orientation ofthe mobile computing device obtained from the accelerometer and compassof the mobile computing device, etc.

In some embodiments, though embodiments are provided that do notmaintain any biometric information about account holders, the system canuse biometric information to assist matching a not-yet-linked trackedsubject to a user account. For example, in one embodiment, the systemstores “hair color” of the customer in his or her user account record.During the matching process, the system might use for example hair colorof subjects as an additional input to disambiguate and match the trackedsubject to a user account. If the user has red colored hair and there isonly one subject with red colored hair in the area of real space or inclose proximity of the mobile computing device, then the system mightselect the subject with red hair color to match the user account. Thedetails of matching engine are presented in U.S. patent application Ser.No. 16/255,573, filed on 23 Jan. 2019, titled, “Systems and Methods toCheck-in Shoppers in a Cashier-less Store” which is incorporated hereinby reference as if fully set forth herein.

The flowcharts in FIGS. 13 to 16C present process steps of fourtechniques usable alone or in combination by the matching engine 170.

Semaphore Images

FIG. 13 is a flowchart 1300 presenting process steps for a firsttechnique to identify subject by matching tracked subjects in the areaof real space with their respective user accounts. In the example of ashopping store, the subjects are customers (or shoppers) moving in thestore in aisles between shelves and other open spaces. The processstarts at step 1302. As a subject enters the area of real space, thesubject opens a client application on a mobile computing device andattempts to login. The system verifies the user credentials at step 1304(for example, by querying the user account database 164) and acceptslogin communication from the client application to associate anauthenticated user account with the mobile computing device. The systemdetermines that the user account of the client application is not yetlinked to a tracked subject. The system sends a semaphore image to theclient application for display on the mobile computing device at step1306. Examples of semaphore images include various shapes of solidcolors such as a red rectangle or a pink elephant, etc. A variety ofimages can be used as semaphores, preferably suited for high confidencerecognition by the image recognition engine. Each semaphore image canhave a unique identifier. The processing system includes logic to acceptlogin communications from a client application on a mobile deviceidentifying a user account before matching the user account to a trackedsubject in the area of real space, and after accepting logincommunications sends a selected semaphore image from the set ofsemaphore images to the client application on the mobile device.

In one embodiment, the system selects an available semaphore image fromthe image database 160 for sending to the client application. Aftersending the semaphore image to the client application, the systemchanges a status of the semaphore image in the image database 166 as“assigned” so that this image is not assigned to any other clientapplication. The status of the image remains as “assigned” until theprocess to match the tracked subject to the mobile computing device iscomplete. After matching is complete, the status can be changed to“available.” This allows for rotating use of a small set of semaphoresin a given system, simplifying the image recognition problem.

The client application receives the semaphore image and displays it onthe mobile computing device. In one embodiment, the client applicationalso increases the brightness of the display to increase the imagevisibility. The image is captured by one or more cameras 114 and sent toan image processing engine, referred to as WhatCNN. The system usesWhatCNN at step 1308 to recognize the semaphore images displayed on themobile computing device. In one embodiment, WhatCNN is a convolutionalneural network trained to process the specified bounding boxes in theimages to generate a classification of hands of the tracked subjects.One trained WhatCNN processes image frames from one camera. In theexample embodiment of the shopping store, for each hand joint in eachimage frame, the WhatCNN identifies whether the hand joint is empty. TheWhatCNN also identifies a semaphore image identifier (in the imagedatabase 166) or an SKU (stock keeping unit) number of the inventoryitem in the hand joint, a confidence value indicating the item in thehand joint is a non-SKU item (i.e. it does not belong to the shoppingstore inventory) and a context of the hand joint location in the imageframe.

As mentioned above, two or more cameras with overlapping fields of viewcapture images of subjects in real space. Joints of a single subject canappear in image frames of multiple cameras in a respective imagechannel. A WhatCNN model per camera identifies semaphore images(displayed on mobile computing devices) in hands (represented by handjoints) of subjects. A coordination logic combines the outputs ofWhatCNN models into a consolidated data structure listing identifiers ofsemaphore images in left hand (referred to as left_hand_classid) andright hand (right_hand_classid) of tracked subjects (step 1310). Thesystem stores this information in a dictionary mapping tracking_id toleft_hand_classid and right_hand_classid along with a timestamp,including locations of the joints in real space. The details of WhatCNNare presented in U.S. patent application Ser. No. 15/907,112, filed 27Feb. 2018, now U.S. Pat. No. 10,133,933, issued on 20 Nov. 2018, titled,“Item Put and Take Detection Using Image Recognition” which isincorporated herein by reference as if fully set forth herein.

At step 1312, the system checks if the semaphore image sent to theclient application is recognized by the WhatCNN by iterating the outputof the WhatCNN models for both hands of all tracked subjects. If thesemaphore image is not recognized, the system sends a reminder at a step1314 to the client application to display the semaphore image on themobile computing device and repeats process steps 1308 to 1312.Otherwise, if the semaphore image is recognized by WhatCNN, the systemmatches a user_account (from the user account database 164) associatedwith the client application to tracking_id (from the subject database150) of the tracked subject holding the mobile computing device (step1316). In one embodiment, the system maintains this mapping(tracking_id-user_account) until the subject is present in the area ofreal space. In one implementation, the system assigns a unique subjectidentifier (e.g., referred to by subject_id) to the identified subjectand stores a mapping of the subject identifier to the tupletracking_id-user_account. The process ends at step 1318.

Service Location

The flowchart 1400 in FIG. 14 presents process steps for a secondtechnique for identifying subjects by matching tracked subjects withuser accounts. This technique uses radio signals emitted by the mobiledevices indicating location of the mobile devices. The process starts atstep 1402, the system accepts login communication from a clientapplication on a mobile computing device as described above in step 1404to link an authenticated user account to the mobile computing device. Atstep 1406, the system receives service location information from themobile devices in the area of real space at regular intervals. In oneembodiment, latitude and longitude coordinates of the mobile computingdevice emitted from a global positioning system (GPS) receiver of themobile computing device are used by the system to determine thelocation. In one embodiment, the service location of the mobilecomputing device obtained from GPS coordinates has an accuracy between 1to 3 meters. In another embodiment, the service location of a mobilecomputing device obtained from GPS coordinates has an accuracy between 1to 5 meters.

Other techniques can be used in combination with the above technique orindependently to determine the service location of the mobile computingdevice. Examples of such techniques include using signal strengths fromdifferent wireless access points (WAP) such as 250 and 252 shown inFIGS. 11 and 12 as an indication of how far the mobile computing deviceis from respective access points. The system then uses known locationsof wireless access points (WAP) 250 and 252 to triangulate and determinethe position of the mobile computing device in the area of real space.Other types of signals (such as Bluetooth, ultra-wideband, and ZigBee)emitted by the mobile computing devices can also be used to determine aservice location of the mobile computing device.

The system monitors the service locations of mobile devices with clientapplications that are not yet linked to a tracked subject at step 1408at regular intervals such as every second. At step 1408, the systemdetermines the distance of a mobile computing device with an unmatcheduser account from all other mobile computing devices with unmatched useraccounts. The system compares this distance with a pre- determinedthreshold distance “d” such as 3 meters. If the mobile computing deviceis away from all other mobile devices with unmatched user accounts by atleast “d” distance (step 1410), the system determines a nearest not yetlinked subject to the mobile computing device (step 1414). The locationof the tracked subject is obtained from the output of the JointsCNN atstep 1412. In one embodiment the location of the subject obtained fromthe JointsCNN is more accurate than the service location of the mobilecomputing device. At step 1416, the system performs the same process asdescribed above in flowchart 1300 to match the tracking_id of thetracked subject with the user_account of the client application. Theprocess ends at a step 1418.

No biometric identifying information is used for matching the trackedsubject with the user account, and none is stored in support of thisprocess. That is, there is no information in the sequences of imagesused to compare with stored biometric information for the purposes ofmatching the tracked subjects with user account in support of thisprocess. Thus, this logic to match the tracked subjects with useraccounts operates without use of personal identifying biometricinformation associated with the user accounts.

Speed and Orientation

The flowchart 1500 in FIG. 15 presents process steps for a thirdtechnique to identify subject by matching tracked subjects with useraccounts. This technique uses signals emitted by an accelerometer of themobile computing devices to match tracked subjects with clientapplications. The process starts at step 1502. The process starts atstep 1504 to accept login communication from the client application asdescribed above in the first and second techniques. At step 1506, thesystem receives signals emitted from the mobile computing devicescarrying data from accelerometers on the mobile computing devices in thearea of real space, which can be sent at regular intervals. At a step1508, the system calculates an average velocity of all mobile computingdevices with unmatched user accounts.

The accelerometers provide acceleration of mobile computing devicesalong the three axes (x, y, z). In one embodiment, the velocity iscalculated by taking the accelerations values at small time intervals(e.g., at every 10 milliseconds) to calculate the current velocity attime “t” i.e., v_(t)=v₀+a_(t), where v₀ is initial velocity. In oneembodiment, the v₀ is initialized as “0” and subsequently, for everytime t+1, v_(t) becomes v₀. The velocities along the three axes are thencombined to determine an overall velocity of the mobile computing deviceat time “t.” Finally at step 808, the system calculates moving averagesof velocities of all mobile computing devices over a larger period oftime such as 3 seconds which is long enough for the walking gait of anaverage person, or over longer periods of time.

At step 1510, the system calculates Euclidean distance (also referred toas L2 norm) between velocities of all pairs of mobile computing deviceswith unmatched client applications to not yet linked tracked subjects.The velocities of subjects are derived from changes in positions oftheir joints with respect to time, obtained from joints analysis andstored in respective subject data structures 320 with timestamps. In oneembodiment, a location of center of mass of each subject is determinedusing the joints analysis. The velocity, or other derivative, of thecenter of mass location data of the subject is used for comparison withvelocities of mobile computing devices. For eachtracking_id-user_account pair, if the value of the Euclidean distancebetween their respective velocities is less than a threshold_0, ascore_counter for the tracking_id- user_account pair is incremented. Theabove process is performed at regular time intervals, thus updating thescore_counter for each tracking_id-user_account pair.

At regular time intervals (e.g., every one second), the system comparesthe score_counter values for pairs of every unmatched user account withevery not yet linked tracked subject (step 1512). If the highest scoreis greater than threshold_1 (step 1514), the system calculates thedifference between the highest score and the second highest score (forpair of same user account with a different subject) at step 1516. If thedifference is greater than threshold_2, the system selects the mappingof user_account to the tracked subject at step 1518 and follows the sameprocess as described above in step 1516. The process ends at a step1520.

In another embodiment, when JointsCNN recognizes a hand holding a mobilecomputing device, the velocity of the hand (of the tracked subject)holding the mobile computing device is used in above process instead ofusing the velocity of the center of mass of the subject. This improvesperformance of the matching algorithm. To determine values of thethresholds (threshold_0, threshold_1, threshold_2), the system usestraining data with labels assigned to the images. During training,various combinations of the threshold values are used and the output ofthe algorithm is matched with ground truth labels of images to determineits performance. The values of thresholds that result in best overallassignment accuracy are selected for use in production (or inference).

No biometric identifying information is used for matching the trackedsubject with the user account, and none is stored in support of thisprocess. That is, there is no information in the sequences of imagesused to compare with stored biometric information for the purposes ofmatching the tracked subjects with user accounts in support of thisprocess. Thus, this logic to match the tracked subjects with useraccounts operates without use of personal identifying biometricinformation associated with the user accounts.

Network Ensemble

A network ensemble is a learning paradigm where many networks arejointly used to solve a problem. Ensembles typically improve theprediction accuracy obtained from a single classifier by a factor thatvalidates the effort and cost associated with learning multiple models.In the fourth technique to match user accounts to not yet linked trackedsubjects, the second and third techniques presented above are jointlyused in an ensemble (or network ensemble). To use the two techniques inan ensemble, relevant features are extracted from application of the twotechniques. FIGS. 16A-16C present process steps (in a flowchart 1600)for extracting features, training the ensemble and using the trainedensemble to predict match of a user account to a not yet linked trackedsubject.

FIG. 16A presents the process steps for generating features using thesecond technique that uses service location of mobile computing devices.The process starts at step 1602. At a step 1604, a Count_X, for thesecond technique is calculated indicating a number of times a servicelocation of a mobile computing device with an unmatched user account isX meters away from all other mobile computing devices with unmatcheduser accounts. At step 1606, Count_X values of all tuples oftracking_id- user_account pairs is stored by the system for use by theensemble. In one embodiment, multiple values of X are used e.g., 1 m, 2m, 3 m, 4 m, 5 m (steps 1608 and 1610). For each value of X, the countis stored as a dictionary that maps tuples of tracking_id- user_accountto count score, which is an integer. In the example where 5 values of Xare used, five such dictionaries are created at step 1612. The processends at step 1614.

FIG. 16B presents the process steps for generating features using thethird technique that uses velocities of mobile computing devices. Theprocess starts at step 1620. At a step 1622, a Count_Y, for the thirdtechnique is determined which is equal to score_counter valuesindicating number of times Euclidean distance between a particulartracking_id-user_account pair is below a threshold_0. At a step 1624,Count_Y values of all tuples of tracking_id-user_account pairs is storedby the system for use by the ensemble. In one embodiment, multiplevalues of threshold _0 are used e.g., five different values (steps 1626and 1628). For each value of threshold_0, the Count_Y is stored as adictionary that maps tuples of tracking_id-user_account to count score,which is an integer. In the example where 5 values of threshold areused, five such dictionaries are created at step 1630. The process endsat step 1632.

The features from the second and third techniques are then used tocreate a labeled training data set and used to train the networkensemble. To collect such a data set, multiple subjects (shoppers) walkin an area of real space such as a shopping store. The images of thesesubject are collected using cameras 114 at regular time intervals. Humanlabelers review the images and assign correct identifiers (tracking_idand user_account) to the images in the training data. The process isdescribed in a flowchart 1600 presented in FIG. 16C. The process startsat a step 1640. At a step 1642, features in the form of Count_X andCount_Y dictionaries obtained from second and third techniques arecompared with corresponding true labels assigned by the human labelerson the images to identify correct matches (true) and incorrect matches(false) of tracking_id and user_account.

As we have only two categories of outcome for each mapping oftracking_id and user_account: true or false, a binary classifier istrained using this training data set (step 1644). Commonly used methodsfor binary classification include decision trees, random forest, neuralnetworks, gradient boost, support vector machines, etc. A trained binaryclassifier is used to categorize new probabilistic observations as trueor false. The trained binary classifier is used in production (orinference) by giving as input Count_X and Count_Y dictionaries fortracking_id-user_account tuples. The trained binary classifierclassifies each tuple as true or false at a step 1646. The process endsat a step 1648.

If there is an unmatched mobile computing device in the area of realspace after application of the above four techniques, the system sends anotification to the mobile computing device to open the clientapplication. If the user accepts the notification, the clientapplication will display a semaphore image as described in the firsttechnique. The system will then follow the steps in the first techniqueto check-in the shopper (match tracking_id to user_account). If thecustomer does not respond to the notification, the system will send anotification to an employee in the shopping store indicating thelocation of the unmatched customer. The employee can then walk to thecustomer, ask him to open the client application on his mobile computingdevice to check-in to the system using a semaphore image.

No biometric identifying information is used for matching the trackedsubject with the user account, and none is stored in support of thisprocess. That is, there is no information in the sequences of imagesused to compare with stored biometric information for the purposes ofmatching the tracked subjects with user accounts in support of thisprocess. Thus, this logic to match the tracked subjects with useraccounts operates without use of personal identifying biometricinformation associated with the user accounts.

Architecture

An example architecture of a system in which the four techniquespresented above are applied to identify subjects by matching auser_account to a not yet linked tracked subject in an area of realspace is presented in FIG. 17. Because FIG. 17 is an architecturaldiagram, certain details are omitted to improve the clarity ofdescription. The system presented in FIG. 17 receives image frames froma plurality of cameras 114. As described above, in one embodiment, thecameras 114 can be synchronized in time with each other, so that imagesare captured at the same time, or close in time, and at the same imagecapture rate. Images captured in all the cameras covering an area ofreal space at the same time, or close in time, are synchronized in thesense that the synchronized images can be identified in the processingengines as representing different views at a moment in time of subjectshaving fixed positions in the real space. The images are stored in acircular buffer of image frames per camera 1702.

A “subject tracking” subsystem 1704 (also referred to as first imageprocessors) processes image frames received from cameras 114 to locateand track subjects in the real space. The first image processors includesubject image recognition engines such as the JointsCNN above.

A “semantic diffing” subsystem 1706 (also referred to as second imageprocessors) includes background image recognition engines, which receivecorresponding sequences of images from the plurality of cameras andrecognize semantically significant differences in the background (i.e.inventory display structures like shelves) as they relate to puts andtakes of inventory items, for example, over time in the images from eachcamera. The second image processors receive output of the subjecttracking subsystem 1704 and image frames from cameras 114 as input.Details of “semantic diffing” subsystem are presented in U.S. patentapplication Ser. No. 15/945,466, filed 4 Apr. 2018, now U.S. Pat. No.10,127,438, issued on 13 Nov. 2018, titled, “Predicting Inventory Eventsusing Semantic Diffing,” and U.S. patent application Ser. No.15/945,473, filed 4 Apr. 2018, titled, “Predicting Inventory Eventsusing Foreground/Background Processing,” both of which are incorporatedherein by reference as if fully set forth herein. The second imageprocessors process identified background changes to make a first set ofdetections of takes of inventory items by tracked subjects and of putsof inventory items on inventory display structures by tracked subjects.The first set of detections are also referred to as backgrounddetections of puts and takes of inventory items. In the example of ashopping store, the first detections identify inventory items taken fromthe shelves or put on the shelves by customers or employees of thestore. The semantic diffing subsystem includes the logic to associateidentified background changes with tracked subjects.

A “region proposals” subsystem 1708 (also referred to as third imageprocessors) includes foreground image recognition engines, receivescorresponding sequences of images from the plurality of cameras 114, andrecognizes semantically significant objects in the foreground (i.e.shoppers, their hands and inventory items) as they relate to puts andtakes of inventory items, for example, over time in the images from eachcamera. The region proposals subsystem 1708 also receives output of thesubject tracking subsystem 1704. The third image processors processsequences of images from cameras 114 to identify and classify foregroundchanges represented in the images in the corresponding sequences ofimages. The third image processors process identified foreground changesto make a second set of detections of takes of inventory items bytracked subjects and of puts of inventory items on inventory displaystructures by tracked subjects. The second set of detections are alsoreferred to as foreground detection of puts and takes of inventoryitems. In the example of a shopping store, the second set of detectionsidentifies takes of inventory items and puts of inventory items oninventory display structures by customers and employees of the store.The details of a region proposal subsystem are presented in U.S. patentapplication Ser. No. 15/907,112, filed 27 Feb. 2018, now U.S. Pat. No.10,133,933, issued on 20 Nov. 2018, titled, “Item Put and Take DetectionUsing Image Recognition” which is incorporated herein by reference as iffully set forth herein.

The system described in FIG. 17 includes a selection logic 1710 toprocess the first and second sets of detections to generate log datastructures including lists of inventory items for tracked subjects. Fora take or put in the real space, the selection logic 1710 selects theoutput from either the semantic diffing subsystem 1706 or the regionproposals subsystem 1708. In one embodiment, the selection logic 1710uses a confidence score generated by the semantic diffing subsystem forthe first set of detections and a confidence score generated by theregion proposals subsystem for a second set of detections to make theselection. The output of the subsystem with a higher confidence scorefor a particular detection is selected and used to generate a log datastructure 1712 (also referred to as a shopping cart data structure)including a list of inventory items (and their quantities) associatedwith tracked subjects.

To process a payment for the items in the log data structure 1712, thesystem in FIG. 17 applies the four techniques for matching the trackedsubject (associated with the log data) to a user_account which includesa payment method such as credit card or bank account information. In oneembodiment, the four techniques are applied sequentially as shown in thefigure. If the process steps in flowchart 1300 for the first techniqueproduces a match between the subject and the user account then thisinformation is used by a payment processor 1736 to charge the customerfor the inventory items in the log data structure. Otherwise (step1728), the process steps presented in flowchart 1400 for the secondtechnique are followed and the user account is used by the paymentprocessor 1736. If the second technique is unable to match the useraccount with a subject (1730) then the process steps presented inflowchart 1500 for the third technique are followed. If the thirdtechnique is unable to match the user account with a subject (1732) thenthe process steps in flowchart 1600 for the fourth technique arefollowed to match the user account with a subject.

If the fourth technique is unable to match the user account with asubject (1734), the system sends a notification to the mobile computingdevice to open the client application and follow the steps presented inthe flowchart 1300 for the first technique. If the customer does notrespond to the notification, the system will send a notification to anemployee in the shopping store indicating the location of the unmatchedcustomer. The employee can then walk to the customer, ask him to openthe client application on his mobile computing device to check-in to thesystem using a semaphore image (step 1740). It is understood that inother embodiments of the architecture presented in FIG. 17, fewer thanfour techniques can be used to match the user accounts to not yet linkedtracked subjects.

Any data structures and code described or referenced above are storedaccording to many implementations in computer readable memory, whichcomprises a non-transitory computer-readable storage medium, which maybe any device or medium that can store code and/or data for use by acomputer system. This includes, but is not limited to, volatile memory,non-volatile memory, application-specific integrated circuits (ASICs),field-programmable gate arrays (FPGAs), magnetic and optical storagedevices such as disk drives, magnetic tape, CDs (compact discs), DVDs(digital versatile discs or digital video discs), or other media capableof storing computer-readable media now known or later developed.

The preceding description is presented to enable the making and use ofthe technology disclosed. Various modifications to the disclosedimplementations will be apparent, and the general principles definedherein may be applied to other implementations and applications withoutdeparting from the spirit and scope of the technology disclosed. Thus,the technology disclosed is not intended to be limited to theimplementations shown, but is to be accorded the widest scope consistentwith the principles and features disclosed herein. The scope of thetechnology disclosed is defined by the appended claims.

What is claimed is:
 1. A system for tracking subjects, in an area ofreal space, comprising: a processing system receiving a plurality ofsequences of images of corresponding fields of view in the real space,including: logic that uses the sequences of images to locate subjects inthe area of real space during identification intervals; and logic thatmatches located subjects from a plurality of identification intervals toidentify tracked subjects, including comparing located subjects in acurrent identification interval with tracked subjects in a firstpreceding identification interval, and when there is a mismatch innumbers of located subjects in the current identification interval withnumbers of tracked subjects in the first preceding identificationinterval, comparing at least one of the located subjects in the currentidentification interval with tracked subjects in a second precedingidentification interval.
 2. The system of claim 1, wherein the systemhas access to data locating places in the area of real spacecorresponding to entrances and exits, when there is a mismatch innumbers of located subjects in the current identification interval withnumbers of tracked subjects in the first preceding identificationinterval, comparing a location of at least one located subject in acurrent identification interval with the places.
 3. The system of claim1, the processing system having access to a database storing a set ofidentified subjects; and logic to link tracked subjects with identifiedsubjects.
 4. The system of claim 1, wherein the comparing includes aprocedure to; detect a condition in which a number of located subjectsin the current identification interval does not match a number oftracked subjects in a first preceding identification interval in theplurality of previous identification intervals; and upon detection ofthe condition, compare at least one of the located subjects in thecurrent set with the set of tracked subjects from a second precedingidentification interval in the plurality of previous identificationintervals, that precedes the first preceding identification interval. 5.The system of claim 4, wherein upon detection of the condition, if theset of tracked subjects from the first preceding identification intervalincludes N members, and the set of tracked subjects from the secondpreceding identification interval includes said N members plus anadditional tracked subject missing in the first preceding identificationinterval, and the current set of located subjects includes said Nmembers plus a candidate located subject, then the procedure includesevaluating the candidate located subject in the current identificationinterval with the tracked subject missing in the second precedingidentification interval.
 6. The system of claim 5, wherein saidevaluating the candidate located subject in the current identificationinterval with the tracked subject missing in the second precedingidentification interval, includes: linking the candidate located subjectand the missing tracked subject if a difference in timestamps betweenthe candidate located subject and the missing tracked subject is lessthan a timing threshold.
 7. The system of claim 5, wherein saidevaluating the candidate located subject in the current identificationinterval with the tracked subject missing in the second precedingidentification interval, includes: linking the candidate located subjectand the missing tracked subject if a distance in real space betweenlocations of the candidate located subject and the missing trackedsubject is less than a distance threshold.
 8. The system of claim 4,wherein the logic that matches located subjects from a plurality ofidentification intervals to identify tracked subjects, logic to simplifylinking N members of the set of located subjects from the currentidentification interval to N members of the set of tracked subjects fromthe first preceding identification interval by prioritizing evaluationof members of the set of located subjects from the currentidentification interval according to locations of the identifiedsubjects.
 9. The system of claim 4, wherein the system has access todata locating places in the area of real space corresponding to adesignated unmonitored location, when there is a mismatch in numbers oflocated subjects in the current identification interval with numbers oftracked subjects in the first preceding identification interval,comparing a location of at least one located subject in a currentidentification interval with the a designated unmonitored location; andlogic to determine that no other subject in the set of tracked subjectsin the second preceding identification interval and the set of locatedsubjects in the current identification interval is closer to thedesignated unmonitored location than a distance threshold.
 10. Thesystem of claim 1, wherein the plurality of sequences of images ofcorresponding fields of view in the area of real space are produced by aplurality of sensors in the area of real space, producing synchronizedsequences of images of respective fields of view.
 11. The system ofclaim 1, wherein the logic that uses the sequences of images to locatesubjects in the area of real space during identification intervalsincludes logic to detect a subject as a constellation of joints in thearea of real space, a joint in the constellation of joints including alocation represented by positions in three dimensions of the area ofreal space and a timestamp.
 12. A method of tracking subjects, in anarea of real space, the method including: receiving a plurality ofsequences of images of corresponding fields of view in the real space;locating subjects in the area of real space using the sequences ofimages during identification intervals; and matching located subjectsfrom a plurality of identification intervals to identify trackedsubjects, including comparing located subjects in a currentidentification interval with tracked subjects in a first precedingidentification interval, and when there is a mismatch in numbers oflocated subjects in the current identification interval with numbers oftracked subjects in the first preceding identification interval,comparing at least one of the located subjects in the currentidentification interval with tracked subjects in a second precedingidentification interval.
 13. The method of claim 12, wherein when thereis a mismatch in numbers of located subjects in the currentidentification interval with numbers of tracked subjects in the firstpreceding identification interval, accessing data locating places in thearea of real space corresponding to entrances and exits, and comparing alocation of at least one located subject in a current identificationinterval with the places.
 14. The method of claim 12, including linkingtracked subjects with identified subjects.
 15. The method of claim 12,wherein the comparing includes a procedure to; detect a condition inwhich a number of located subjects in the current identificationinterval does not match a number of tracked subjects in a firstpreceding identification interval in the plurality of previousidentification intervals; and upon detection of the condition, compareat least one of the located subjects in the current set with the set oftracked subjects from a second preceding identification interval in theplurality of previous identification intervals, that precedes the firstpreceding identification interval.
 16. The method of claim 15, whereinupon detection of the condition, if the set of tracked subjects from thefirst preceding identification interval includes N members, and the setof tracked subjects from the second preceding identification intervalincludes said N members plus an additional tracked subject missing inthe first preceding identification interval, and the current set oflocated subjects includes said N members plus a candidate locatedsubject, then the procedure includes evaluating the candidate locatedsubject in the current identification interval with the tracked subjectmissing in the second preceding identification interval.
 17. The methodof claim 16, wherein said evaluating the candidate located subject inthe current identification interval with the tracked subject missing inthe second preceding identification interval, includes: linking thecandidate located subject and the missing tracked subject if adifference in timestamps between the candidate located subject and themissing tracked subject is less than a timing threshold.
 18. The methodof claim 16, wherein said evaluating the candidate located subject inthe current identification interval with the tracked subject missing inthe second preceding identification interval, includes: linking thecandidate located subject and the missing tracked subject if a distancein real space between locations of the candidate located subject and themissing tracked subject is less than a distance threshold.
 19. Themethod of claim 15, including linking N members of the set of locatedsubjects from the current identification interval to N members of theset of tracked subjects from the first preceding identification intervalby prioritizing evaluation of members of the set of located subjectsfrom the current identification interval according to locations of theidentified subjects.
 20. The method of claim 15, wherein when there is amismatch in numbers of located subjects in the current identificationinterval with numbers of tracked subjects in the first precedingidentification interval, accessing data locating places in the area ofreal space corresponding to a designated unmonitored location, andcomparing a location of at least one located subject in a currentidentification interval with the a designated unmonitored location; anddetermining that no other subject in the set of tracked subjects in thesecond preceding identification interval and the set of located subjectsin the current identification interval is closer to the designatedunmonitored location than a distance threshold.
 21. The method of claim12, wherein the plurality of sequences of images of corresponding fieldsof view in the area of real space are produced by a plurality of sensorsin the area of real space, producing synchronized sequences of images ofrespective fields of view.
 22. The method of claim 12, includinglocating subjects in the area of real space using the sequences ofimages during identification intervals by detecting a subject as aconstellation of joints in the area of real space, a joint in theconstellation of joints including a location represented by positions inthree dimensions of the area of real space and a timestamp.
 23. Anon-transitory computer readable storage medium impressed with computerprogram instructions to track subjects, in an area of real space, theinstructions, when executed on a processor, implement a methodcomprising: receiving a plurality of sequences of images ofcorresponding fields of view in the real space; locating subjects in thearea of real space using the sequences of images during identificationintervals; and matching located subjects from a plurality ofidentification intervals to identify tracked subjects, includingcomparing located subjects in a current identification interval withtracked subjects in a first preceding identification interval, and whenthere is a mismatch in numbers of located subjects in the currentidentification interval with numbers of tracked subjects in the firstpreceding identification interval, comparing at least one of the locatedsubjects in the current identification interval with tracked subjects ina second preceding identification interval.
 24. The storage medium ofclaim 23, wherein when there is a mismatch in numbers of locatedsubjects in the current identification interval with numbers of trackedsubjects in the first preceding identification interval, accessing datalocating places in the area of real space corresponding to entrances andexits, and comparing a location of at least one located subject in acurrent identification interval with the places.
 25. The storage mediumof claim 23, the method including linking tracked subjects withidentified subjects.
 26. The storage medium of claim 23, wherein thecomparing includes a procedure to; detect a condition in which a numberof located subjects in the current identification interval does notmatch a number of tracked subjects in a first preceding identificationinterval in the plurality of previous identification intervals; and upondetection of the condition, compare at least one of the located subjectsin the current set with the set of tracked subjects from a secondpreceding identification interval in the plurality of previousidentification intervals, that precedes the first precedingidentification interval.
 27. The storage medium of claim 26, wherein themethod includes upon detection of the condition, if the set of trackedsubjects from the first preceding identification interval includes Nmembers, and the set of tracked subjects from the second precedingidentification interval includes said N members plus an additionaltracked subject missing in the first preceding identification interval,and the current set of located subjects includes said N members plus acandidate located subject, then the procedure includes evaluating thecandidate located subject in the current identification interval withthe tracked subject missing in the second preceding identificationinterval.
 28. The storage medium of claim 27, wherein said evaluatingthe candidate located subject in the current identification intervalwith the tracked subject missing in the second preceding identificationinterval, includes: linking the candidate located subject and themissing tracked subject if a difference in timestamps between thecandidate located subject and the missing tracked subject is less than atiming threshold.
 29. The storage medium of claim 27, wherein saidevaluating the candidate located subject in the current identificationinterval with the tracked subject missing in the second precedingidentification interval, includes: linking the candidate located subjectand the missing tracked subject if a distance in real space betweenlocations of the candidate located subject and the missing trackedsubject is less than a distance threshold.
 30. The storage medium ofclaim 26, the method including linking N members of the set of locatedsubjects from the current identification interval to N members of theset of tracked subjects from the first preceding identification intervalby prioritizing evaluation of members of the set of located subjectsfrom the current identification interval according to locations of theidentified subjects.
 31. The storage medium of claim 26, wherein themethod includes, when there is a mismatch in numbers of located subjectsin the current identification interval with numbers of tracked subjectsin the first preceding identification interval, accessing data locatingplaces in the area of real space corresponding to a designatedunmonitored location, and comparing a location of at least one locatedsubject in a current identification interval with the a designatedunmonitored location; and determining that no other subject in the setof tracked subjects in the second preceding identification interval andthe set of located subjects in the current identification interval iscloser to the designated unmonitored location than a distance threshold.32. The storage medium of claim 23, wherein the plurality of sequencesof images of corresponding fields of view in the area of real space areproduced by a plurality of sensors in the area of real space, producingsynchronized sequences of images of respective fields of view.
 33. Thestorage medium of claim 23, wherein the method includes locatingsubjects in the area of real space using the sequences of images duringidentification intervals by detecting a subject as a constellation ofjoints in the area of real space, a joint in the constellation of jointsincluding a location represented by positions in three dimensions of thearea of real space and a timestamp.
 34. A system for tracking subjectsin an area of real space, comprising: a processing system receiving aplurality of sequences of images of corresponding fields of view in thereal space, including: logic that uses the sequences of images to locatesubjects in the area of real space during identification intervals;logic that matches located subjects from a plurality of identificationintervals to identify tracked subjects; logic that links trackedsubjects to user accounts, to maintain a record of identified subjectsin the area of real space; and logic that compares located subjects in acurrent identification interval with identified subjects in a pluralityof preceding identification intervals to update locations of theidentified subjects, including when there is a mismatch in numbers oflocated subjects in the current identification interval with numbers ofidentified subjects in a first preceding identification interval, thencomparing at least one of the located subjects in the currentidentification interval with identified subjects in at least one moreother preceding identification interval.
 35. The system of claim 34,wherein the logic to link tracked subjects with user accounts operateswithout personal biometric information about users.
 36. The system ofclaim 34, wherein the logic to link tracked subjects with user accountsincludes a procedure to identify locations of mobile devices executingclient applications in the area of real space, and matching locations ofthe mobile devices with locations of the tracked or located subjects.37. A method tracking subjects in an area of real space, comprising:receiving a plurality of sequences of images of corresponding fields ofview in the real space; using the sequences of images to locate subjectsin the area of real space during identification intervals; matchinglocated subjects from a plurality of identification intervals toidentify tracked subjects; linking tracked subjects to user accounts, tomaintain a record of identified subjects in the area of real space; andcomparing located subjects in a current identification interval withidentified subjects in a plurality of preceding identification intervalsto update locations of the identified subjects, including when there isa mismatch in numbers of located subjects in the current identificationinterval with numbers of identified subjects in a first precedingidentification interval, then comparing at least one of the locatedsubjects in the current identification interval with identified subjectsin at least one more other preceding identification interval.
 38. Themethod of claim 37, wherein linking tracked subjects with user accountsoperates without personal biometric information about users.
 39. Themethod of claim 37, wherein linking tracked subjects with user accountsincludes a procedure to identify locations of mobile devices executingclient applications in the area of real space, and matching locations ofthe mobile devices with locations of the tracked or located subjects.40. A non-transitory computer readable storage medium impressed withcomputer program instructions to track subjects, in an area of realspace, the instructions, when executed on a processor, implement amethod comprising: receiving a plurality of sequences of images ofcorresponding fields of view in the real space; using the sequences ofimages to locate subjects in the area of real space duringidentification intervals; matching located subjects from a plurality ofidentification intervals to identify tracked subjects; linking trackedsubjects to user accounts, to maintain a record of identified subjectsin the area of real space; and comparing located subjects in a currentidentification interval with identified subjects in a plurality ofpreceding identification intervals to update locations of the identifiedsubjects, including when there is a mismatch in numbers of locatedsubjects in the current identification interval with numbers ofidentified subjects in a first preceding identification interval, thencomparing at least one of the located subjects in the currentidentification interval with identified subjects in at least one moreother preceding identification interval.
 41. The storage medium of claim40, wherein linking tracked subjects with user accounts operates withoutpersonal biometric information about users.
 42. The storage medium ofclaim 40, wherein linking tracked subjects with user accounts includes aprocedure to identify locations of mobile devices executing clientapplications in the area of real space, and matching locations of themobile devices with locations of the tracked or located subjects.